返回信息流问题如题，我想增加每个任务的内存限制，但是不知道怎么正确配置，希望有人可以帮助我。
## hadoop/conf
core_site.xml 中，我设置了
```
<property>
<name>io.sort.mb</name>
<value>3000</value>
</property>
<property>
<name>io.sort.factor</name>
<value>300</value>
</property>
```
## eclipse
在 Edit Hadoop location 的 Advanced parameters 中，我也配置 io.sort.factor 和 io.sort.mb ，但是运行时还是默认的 100
## 运行结果
```
17/04/07 12:43:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/07 12:43:22 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
17/04/07 12:43:22 INFO input.FileInputFormat: Total input paths to process : 1
17/04/07 12:43:23 INFO mapred.JobClient: Running job: job_local805018878_0001
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Waiting for map tasks
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:23 INFO util.ProcessTree: setsid exited with exit code 0
17/04/07 12:43:23 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@58948e69
17/04/07 12:43:23 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:0+67108864
17/04/07 12:43:23 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:23 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:23 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:24 INFO mapred.JobClient: map 0% reduce 0%
17/04/07 12:43:25 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:25 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:25 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:25 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:27 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:27 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:27 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:27 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:28 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:28 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:28 INFO mapred.Task: Task:attempt_local805018878_0001_m_000000_0 is done. And is in the process of commiting
17/04/07 12:43:28 INFO mapred.LocalJobRunner:
17/04/07 12:43:28 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000000_0' done.
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:28 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@742efee7
17/04/07 12:43:28 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:67108864+67108864
17/04/07 12:43:28 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:28 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:28 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:29 INFO mapred.JobClient: map 20% reduce 0%
17/04/07 12:43:29 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:29 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:29 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:29 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:31 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:31 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:31 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:31 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:31 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:31 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:31 INFO mapred.Task: Task:attempt_local805018878_0001_m_000001_0 is done. And is in the process of commiting
17/04/07 12:43:31 INFO mapred.LocalJobRunner:
17/04/07 12:43:31 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000001_0' done.
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:31 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@42c39ed
17/04/07 12:43:31 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:134217728+67108864
17/04/07 12:43:31 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:31 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:31 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:32 INFO mapred.JobClient: map 40% reduce 0%
17/04/07 12:43:33 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:33 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:33 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:33 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:34 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:34 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:34 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:34 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:34 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:34 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:34 INFO mapred.Task: Task:attempt_local805018878_0001_m_000002_0 is done. And is in the process of commiting
17/04/07 12:43:34 INFO mapred.LocalJobRunner:
17/04/07 12:43:34 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000002_0' done.
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:34 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@16b56343
17/04/07 12:43:34 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:201326592+67108864
17/04/07 12:43:34 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:34 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:34 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:35 INFO mapred.JobClient: map 60% reduce 0%
17/04/07 12:43:36 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:36 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:36 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:36 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:37 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:37 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:37 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:37 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:37 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:37 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:37 INFO mapred.Task: Task:attempt_local805018878_0001_m_000003_0 is done. And is in the process of commiting
17/04/07 12:43:37 INFO mapred.LocalJobRunner:
17/04/07 12:43:37 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000003_0' done.
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:37 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4b9c21a4
17/04/07 12:43:37 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:268435456+67108864
17/04/07 12:43:37 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:37 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:37 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:38 INFO mapred.JobClient: map 80% reduce 0%
17/04/07 12:43:38 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:38 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:38 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:38 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:40 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:40 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:40 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:40 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:40 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:40 INFO mapred.Task: Task:attempt_local805018878_0001_m_000004_0 is done. And is in the process of commiting
17/04/07 12:43:40 INFO mapred.LocalJobRunner:
17/04/07 12:43:40 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000004_0' done.
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Map task executor complete.
17/04/07 12:43:40 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@24feb67c
17/04/07 12:43:40 INFO mapred.LocalJobRunner:
17/04/07 12:43:40 INFO mapred.Merger: Merging 5 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 47185930 bytes
17/04/07 12:43:40 INFO mapred.LocalJobRunner:
17/04/07 12:43:41 INFO mapred.JobClient: map 100% reduce 0%
17/04/07 12:43:43 INFO mapred.Task: Task:attempt_local805018878_0001_r_000000_0 is done. And is in the process of commiting
17/04/07 12:43:43 INFO mapred.LocalJobRunner:
17/04/07 12:43:43 INFO mapred.Task: Task attempt_local805018878_0001_r_000000_0 is allowed to commit now
17/04/07 12:43:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local805018878_0001_r_000000_0' to hdfs://localhost:9000/user/ubuntu/output
17/04/07 12:43:43 INFO mapred.LocalJobRunner: reduce > reduce
17/04/07 12:43:43 INFO mapred.Task: Task 'attempt_local805018878_0001_r_000000_0' done.
17/04/07 12:43:44 INFO mapred.JobClient: map 100% reduce 100%
17/04/07 12:43:44 INFO mapred.JobClient: Job complete: job_local805018878_0001
17/04/07 12:43:44 INFO mapred.JobClient: Counters: 22
17/04/07 12:43:44 INFO mapred.JobClient: Map-Reduce Framework
17/04/07 12:43:44 INFO mapred.JobClient: Spilled Records=7864320
17/04/07 12:43:44 INFO mapred.JobClient: Map output materialized bytes=47185950
17/04/07 12:43:44 INFO mapred.JobClient: Reduce input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient: Map input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient: SPLIT_RAW_BYTES=650
17/04/07 12:43:44 INFO mapred.JobClient: Map output bytes=41943040
17/04/07 12:43:44 INFO mapred.JobClient: Reduce shuffle bytes=0
17/04/07 12:43:44 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient: Reduce input groups=2621440
17/04/07 12:43:44 INFO mapred.JobClient: Combine output records=0
17/04/07 12:43:44 INFO mapred.JobClient: Reduce output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient: Map output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient: Combine input records=0
17/04/07 12:43:44 INFO mapred.JobClient: CPU time spent (ms)=0
17/04/07 12:43:44 INFO mapred.JobClient: Total committed heap usage (bytes)=24697110528
17/04/07 12:43:44 INFO mapred.JobClient: File Input Format Counters
17/04/07 12:43:44 INFO mapred.JobClient: Bytes Read=335544320
17/04/07 12:43:44 INFO mapred.JobClient: FileSystemCounters
17/04/07 12:43:44 INFO mapred.JobClient: HDFS_BYTES_READ=1342177280
17/04/07 12:43:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=377897656
17/04/07 12:43:44 INFO mapred.JobClient: FILE_BYTES_READ=235943170
17/04/07 12:43:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=33910562
17/04/07 12:43:44 INFO mapred.JobClient: File Output Format Counters
17/04/07 12:43:44 INFO mapred.JobClient: Bytes Written=33910562
```
这是一条镜像帖。来源：北邮人论坛 / ml-dm / #23528同步于 2017/4/7
ML_DM机器人发帖
使用 eclipse 开发 hadoop1.2.1 时， io.sort.mp 怎么更改？
hmy
2017/4/7镜像同步0 回复
订阅后，新回复会通过你的通知中心匿名送达。
0 条回复
暂无回复 · 你可以订阅本帖等待新回复。