Michael G. Noll
My digital moleskine

1. Doug Cutting Sep 21.07 / 8pm

   This looks great! Is there any chance you could add it to Hadoop’s wiki? Also, it would be great to have your Ubuntu tutorials there too.
2. Berlin Oct 09.07 / 8pm

   I am sorry, I have a couple of questions; in this command, is the actual “mapper.py” script launched by Java? for example, it could be any script, mapper.rb? mapper.bin?

   bin/hadoop jar contrib/hadoop-streaming.jar -mapper /home/hadoop/mapper.py -reducer /home/hadoop/reducer.py -input gutenberg/* -output gutenberg-output

   Also, is there some special format that mapper and reducer have to adhere to?
3. Michael G. Noll Oct 10.07 / 8am

   Hi Berlin!

   Yes, you can use any script as Mapper or Reducer if used in combination with HadoopStreaming (at least that’s my impression; feel free to correct me if I’m wrong and there are some hidden restrictions). If you take a look at the HadoopStreaming documentation on the Hadoop wiki, you will see they use a Perl script as “mapper” in the example at the bottom.

   When a MapReduce job is run with HadoopStreaming, Hadoop will “pack” the specified scripts/files into a temporary job JAR file (JAR format is used here more as a means of file transportation, not in terms of “Java should directly execute it as if it were Java code”) whose contents are finally run somewhere down the processing chain. In other words: Hadoop packs the required files into a temporary archive, moves it somewhere into its hadoop.tmp.dir directory and runs the specified mapper and reduce script similarly to how you would run them in a shell. You can specify which files should be “shipped” to Hadoop via the “-file” parameter, but Hadoop is also clever enough to include “-mapper” and “-reducer” scripts as in my Python MapReduce tutorial. In the Perl example on the wiki, the “-mapper” parameter is actually launching a non-shipped (i.e. somewhere on the normal filesystem) Perl interpreter which in turn launches a shipped custom Perl script.

   Regarding your question about special formats for mapper and reducer, I’m not sure what you mean by “format”. Mapper and reducer scripts only have to read from STDIN (standard input) and write to STDOUT (standard output) in a Hadoop/MapReduce-compliant way. Compliant basically means line-based, i.e. input and output “entries” should be individually read and written as lines. Of course, you can output multiple lines per each input line in the Map step, for instance.

   Hope this helps, Michael
4. Michael G. Noll Oct 10.07 / 4pm

   Oh, and in case you were wondering: I already replied via email to Doug Cutting that I will integrate the tutorials to Hadoop’s official wiki in the next days. I’ve been quite busy lately, so I haven’t had the time yet.
5. Chandra Tambrin Oct 17.07 / 9am

   Hi Michael,

   Thanks for your nice article. I have one question after reading your article. If we specify python command as you explained to Berlin, Hadoop will “pack” the specified scripts/files into a temporary job JAR file. Now if we are on the Multi-Node cluster, will the script be copied to all nodes? and does it mean that all nodes must have the same python (and same libraries) installed? Or is it smart enough to parse the needed library to all nodes (which I highly doubt is the case)?

   This actually leads me to think that if same python (or any other language) need to be installed on each node, then any major upgrade to python, it will require a re-installation to all nodes; am I making a right assumption?

   Thanks, Chandra
6. Michael G. Noll Oct 17.07 / 11am

   Hi Chandra!

   Now if we are on the Multi-Node cluster, will the script be copied to all nodes? and does it mean that all nodes must have the same python (and same libraries) installed? Or is it smart enough to parse the needed library to all nodes (which I highly doubt is the case)?

   From what I know, you – in the sense of “you the cluster operator” – have to make sure that the packed scripts will run as expected on all machines in a multi-node cluster. Hadoop will “only” take care of distributing your packed scripts to all machines on which a job shall run, i.e. to all relevant task trackers.

   This actually leads me to think that if same python (or any other language) need to be installed on each node, then any major upgrade to python, it will require a re-installation to all nodes; am I making a right assumption?

   If the scripts/libraries you want to use for your MapReduce program are affected by such a Python upgrade, yes. On the other hand, I’ve seen a lot of my own code written for Python 2.4 to run without any changes on Python 2.5 – so my answer would be a clear “it depends”

   Generally speaking, you’re absolutely right though. Any software update which introduces some kind of backward compatibility problem or the like might force you to update all the nodes in the cluster. Whether and how you can mitigate these problems in practice depends on your context. For instance, you can prepare your code by using/not using concepts such as __future__, or “import” required functionality by checking for the appropriate libraries depending on the specific Python version on the node*, etc. This can help you to extend the time window until you really must upgrade the cluster.

   Another neat though not directly related trick when upgrading an Hadoop cluster (in particular the Hadoop software itself) is to give updated nodes different/new port numbers for running task trackers etc. than before. This will prevent outdated nodes to connect to your name node and mess up your cluster.

   From my own pragmatic view, I decided to run Ubuntu Linux on all nodes because Ubuntu is so easy to manage due to tools like apt-get. This helps us to get around a lot of these tricky detail problems because it’s much easier to keep all nodes in sync. What also helps is not to blindly rush into new software updates if you don’t really need them but I guess that’s just common sense.

   Hope this helps, Michael

   *for example, some libraries might have been external third-party modules in Python version X but have been integrated to the core modules for Python version X+1; you can often easily handle such cases dynamically in your code
7. Tomasz Oct 31.07 / 10pm

   Micheal,
   credits for the great work you’re doing.

   There is one issue, though, I’m not sure about:

   The mapper from your tutorial does two things: tokenize lines:
   1) words = filter(lambda word: word, line.split())

   and aggregate counts for individual tokens:

   2) word2count[word] = word2count.get(word, 0) + 1

   In this second step it, sort of, duplicates the reducer’s task of
   collecting the counts:
   word2count[word] = word2count.get(word, 0) + count

   If you ran the test on a single file (no hadoop streaming):
   cat … | /home/hadoop/mapper.py | /home/hadoop/reducer.py

   it would give exactly the same output as when running the mapper only:
   cat … | /home/hadoop/mapper.py

   I’m not sure if collecting the counts in a hash twice would be optimal.

   Now, I understand that when working with multiple files (or when portioning the text input, as hadoop does) it would still make sense to split the tasks in this way.

   What do you think? Are there any rules how to optimally split tasks between mapper and reducer? Should they do more or less the same amount of work? Notice, that the mapper’s task would become much more intensive if we wanted to do more advanced tokenization (e.g. separate punctuation from words, recognize urls, numeric expressions, etc.)

   Thanks,

   Tomasz
8. Michael G. Noll Nov 01.07 / 12pm

   Hi Thomasz,

   you’re right that running the pseudo MapReduce runs “cat FILE | ./mapper.py” and “cat FILE | ./mapper.py | ./reducer.py” will give the same results because here, the “setup” is effectively 1 Tasktracker configured to run only 1 map task per job and 1 reduce task per job – similar to having only 1 Tasktracker node in the cluster and setting the Hadoop configuration options “mapred.map.tasks” and “mapred.reduce.tasks” to 1.

   The purpose of the pseudo MapReduce runs is a very basic test to see if your code behaves as expected. It will however not simulate how the actual input and output data is split and shared between multiple nodes when you would finally run in on a real Hadoop cluster. (I re-state this here just for clarity)

   Now coming to your question:

   I’m not sure if collecting the counts in a hash twice would be optimal.

   This depends on what you want to do and how your data looks like. In my tutorial example, yes, you could as well skip hashing in the Map step and simply output “ 1″ for each word found in the input data.

   So, instead of giving

   foo 3
   bar 2

   the Map script could as well output

   foo 1
   foo 1
   bar 1
   foo 1
   bar 1

   and let just the Reduce script compute the final sum of each word’s count(s).

   Now when you run a “real” Hadoop MapReduce job, you might have to decide depending on the context whether you want the Map step to do already part of the “summing” work or not – maybe you’re happy to trade memory/CPU time for less output data to subsequent Reduce steps.

   Having that said, I think I should modify the Map script in the tutorial so that it does not compute sums for its own word counts, in particular because new Hadoop users will run the pseudo examples (“cat FILE | …”) and wonder what the Reduce script actually contributes This might be better on a didactic level. Thanks for pointing this out!

   A related and very interesting discussion on the Hadoop mailing list was about how to handle large Reduce group sizes. Here’s a snippet:

   The values to Reduce is an disk backed iterator. The problematic part is to compute the distinct count. You have to keep the unique values in memory, or you have to use some other tricks.

   Now to your other questions:

   What do you think? Are there any rules how to optimally split tasks between mapper and reducer? Should they do more or less the same amount of work? Notice, that the mapper’s task would become much more intensive if we wanted to do more advanced tokenization (e.g. separate punctuation from words, recognize urls, numeric expressions, etc.)

   Well, I don’t think I can give you a generally correct answer. My own opinion is that there is no free lunch.

   For example, I have run several larger MapReduce jobs in the past where the Map step was just a passthrough, i.e. the output was exactly the input. I had to do this because at that time there was a nasty Hadoop bug that sometimes caused the Reduce step to hang even though the Map step was 100% done.

   In another scenario, we had to use a third-party library which we found to have serious memory leaks. One of the workarounds we were forced to use was (next to limiting available system resources with tools like ulimit etc.) limiting the amount of work in the Map and Reduce steps, and just iteratively run multiple MapReduce jobs one after the other (output of job #1 is input for job #2).

   Admitted, the two examples above are not the general case. When I “design” a MapReduce job, I try to think about what each step will do, what its requirements and effects are (CPU time, memory, local I/O, inter-node I/O, network latency to the outside – whatever you need to do) and then decide how I will setup the work load distribution.

   In practice, I found the passthrough idea (output of one step is just its input) to be extremely helpful to test and harden scripts with regard to error handling etc. For instance, I would just run the Map step to identify errors which might only (statistically) occur in a large-scale scenario but which were rarely seen in small local test runs. After fixing the problematic code, I would store the Map output and use it as passthrough input-output to debug the Reduce step in the same way. And yes, I often found “unexpected” errors in very creative places

   These are my own 2 cents. I’m still learning how to (ab)use Hadoop to help me with my tasks.

   Hope this helps, Michael
9. Justin Mar 24.08 / 4pm

   The reducer should be implemented like this:
   (obviously with correct formatting:))

   #!/usr/bin/env python
   import sys
   from itertools import groupby
   from operator import itemgetter

   def get_file(f):
   for line in f:
   yield line.split(“\t”, 1)

   def main():
   data = get_file(sys.stdin)
   for word, counts in groupby(data, itemgetter(0)):
   tot = sum(int(x[1]) for x in counts)
   print “%s\t%d” % (word, tot)

   if __name__ == “__main__”:
   main()
10. Michael G. Noll Apr 10.08 / 8pm

    Hello Justin, thanks for your good suggestion and sorry for taking so long to respond. I’ve been preparing the improvements for the tutorial but just haven’t found the time to publish them. I will do so in the next days!
11. Michael G. Noll Apr 11.08 / 6pm

    I have finally added improved Mapper and Reduce code to the tutorial. Thanks to all for your suggestions and feedback!
12. Justin Apr 17.08 / 3pm

    Cool.. I started writing a class to abstract out the hadoop stuff, so writing a reduce function only has to handle the key, and the values, just like in the real api.
    I haven’t had much time to fix it up, but it should be useful.. it is only 20 lines of code
13. Vasudev Ram Jul 20.08 / 6am

    Interesting article.

    Maybe mapper.py can be slightly simplified by replacing this:

    words = filter(lambda word: word, line.split())

    with:

    words = line.split()

    Tested only slightly, could be wrong …

    - Vasudev
14. Michael G. Noll Jul 24.08 / 3pm

    Hello Vasudev, you’re right with your suggestion.

    I have spent a long time to figure out why I initially chose to use the line with filter() but I could not remember the reason

    Maybe it was related to filtering out specific words during some test runs, like filter(lambda word: len(word) >= min_length, ...).

    I updated the tutorial to use the shorter version without filter().

    Thanks, Michael
15. Vasudev Ram Aug 14.08 / 6am

    Hello Michael,

    (Sorry for the delay in replying – keeping track of all the blog posts I comment on can get a little difficult sometimes, depending on the frequency with which I post comments.)

    Glad you found my suggestion of help.

    Vasudev
16. Ben May 12.09 / 12am

    Brilliant collection of articles. I have followed them through and set up a 3 pc cluster. Sucessfully ran the java examples then turned my mind to this python one… this is where I am getting issues. Specificly the one below…

    In the below case I was using a shell script which just passed through the data (to test) but I got the same error when trying to run Python code as well. Any ideas? I did some googling but couldnt find any solution to the problem :S

    java.lang.RuntimeException: configuration exception at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:182)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:63)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)
    Caused by: java.io.IOException: Cannot run program “/home/hadoop/mapper.sh”: java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
    … 8 more
    Caused by: java.io.IOException: java.io.IOException: error=2, No such file
    or directory at java.lang.UNIXProcess.(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
    … 9 more
17. Michael G. Noll May 12.09 / 9am

    Hello Ben, have you already looked at this line in your error output?

    Cannot run program “/home/hadoop/mapper.sh”: java.io.IOException: error=2, No such file or directory
18. Ben May 13.09 / 1am

    Yes… which is odd because I am running under the hadoop user and can conform that that file is indeed where it is looking for it.
19. Michael G. Noll May 13.09 / 8am

    Is mapper.sh executable? If not, do

    chmod 777 /home/hadoop/mapper.sh

    and try again. The Hadoop error message does not directly indicate a permission problem, but maybe that’s still the cause of the error.
20. Ben May 14.09 / 10am

    Ahh chmod 777 fixed the trick. Although to get Python scripts to run I also had to write the following,

    /usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-0.19.1-streaming.jar -mapper “python /home/hadoop/map_ users.py” -reducer “python /home/hadoop/reduce_users.py” -input trainingset/* -output trainingset-.output.5

    Not sure if it is using the cluster yet though it looks like it is just running on the master node.
21. Michael G. Noll May 14.09 / 10am

    Hmm, interesting. Do you have a correct shebang line at the start of your Python scripts? Something like:

    #!/usr/bin/env python

    or

    #!/usr/bin/python

    Using the full cluster:

    Normally, Hadoop only activates nodes when it thinks they’re really needed. So if your input problem is very small, Hadoop might use only a subset of all cluster nodes. This might be the reason that only one node is running on your test job.
22. Ben May 14.09 / 11am

    I had a play with it then. Looks like the python script needs to exist in the same location on each node. Turns out I cant seem to just run the python script on my setup. So the above command makes sense from that point of view.

    In any case I am sure I can figure it out from here. Thank you so much for the wonderful article.
23. Michael G. Noll May 14.09 / 12pm

    You’re welcome.

    By the way, there is also some related information on the Hadoop wiki: HadoopStreaming. There is a note that you shouldn’t expect shebangs to work (though I personally never ran into trouble).

    You might also want to add the -file option to your command, e.g.:

    /usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-0.19.1-streaming.jar -file /home/hadoop/map_users.py -mapper “/path/to/python map_ users.py” -file /home/hadoop/reduce_users.py -reducer “/path/to/python reduce_users.py” -input trainingset/* -output trainingset-.output.5
24. Ben May 14.09 / 12pm

    In the end it came down to windows/dos text files. I was editing them on a windows machine and copied them in. The shebangs then didnt work and the whole thing failed. Its all good now though, with the job being passed around correctly.

    Now I just need to build a real cluster to process some of the jobs I have lying around.
25. George May 19.09 / 8pm

    Hi Micheal,

    When I run an example after install hadoop, my command is as below:
    hadoop jar hadoop-0.18.3-example.jar wordcount gutenburg gutenberg-out

    But what I get next is: java.io.IOExcption: Error opening job jar: hadoop-0.18.3-examples.jar
    at org.apache.hadoop.util.Runjar.main(Runjar.java:90)
    ……..

    Why can’t hadoop find the example.jar? I use java, not python.

    Thank you very much!

    George
26. George May 19.09 / 8pm

    I got it. I should go one level up. My example.jar is not located under bin.
27. George May 19.09 / 8pm

    Hi Micheal,

    Do you use eclipse to program map-reduce? I tried IBM plug in for hadoop and can’t even succeed connecting to the hadoop server. Hadoop is installed ok and able to run example and give an output.
    My eclipse version: 3.4.2
    hadoop version: 0.18.3
    Thank you!!
    George
28. Michael G. Noll May 19.09 / 9pm

    Hello George, I don’t use Eclipse for my Hadoop tasks so unfortunately I can’t help you out there.
29. George May 19.09 / 10pm

    Hi Micheal,
    If so what’s your favorite IDE fro the map-reduce programming?

    Thanks!
    George
30. Michael G. Noll May 20.09 / 12pm

    I like to use vi / vim for that.
31. Alex Aug 02.09 / 5pm

    What would be the advantages of using the python streams as you describe in the tutorial instead of the ‘dumbo’ python API http//wiki.github.com/klbostee/dumbo

    Is there any performance impact or is that just all about personal preferences and flexibility?
    Cheers
    Alex
32. Michael G. Noll Aug 03.09 / 9am

    Hello Alex, I don’t have much experience with Dumbo — I tried it once but that was one of the earliest versions of it. I think Klaas’ Dumbo is a great project that will allow even more people to work with Hadoop.

    My personal reasons why I haven’t followed its development that closely: I don’t want another abstraction layer between Hadoop and my own code. I also know what I can expect from Hadoop’s streaming API and how I can (ab)use it.

    Additionally, it seems that Dumbo requires the use of Hadoop patches to make it work properly with Hadoop (please correct me if I’m wrong). Even if it did not, my impression is that upgrading Hadoop might break your MapReduce code because of compatibility issues between Hadoop and Dumbo.

    That said, I’d still consider this a matter of personal preferences. If I had the experience and success with Dumbo, I’d stick to it. Why not?
33. John Adams Sep 05.09 / 3am

    Hi Michael,

    First of all, thank you for the wonderful article. This is my “Hello Hadoop”

    I have a basic question: Is it even possible to implement k-means using the Hadoop Streaming approach? From what I understand, k-means in Hadoop requires a driver program (which I read somewhere else) but I am only able to provide a mapper a reducer. Would you suggest a rough approach for this?

    Thanks,
    John
34. Onur Oct 20.09 / 3am

    Thank you Mike for the great tutorials. Just an informational note for newbies like me. The python example did not work for me (using hadoop 0.20.1) until I used the -file option as:

    ./bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar -file ./mapper.py -mapper ./mapper.py -file ./reducer.py -reducer ./reducer.py -input gutenberg/* -output gutenberg-output2
35. yucc Oct 20.09 / 11am

    I had the same “java.lang.RuntimeException: Error in configuring object” problem too

    I use ruby:
    $ bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar -input input/file1.txt -output output -file sample/mapper.rb -mapper “sample/mapper.rb” -file sample/reducer.rb -reducer “sample/reducer.rb”

    And the log says:
    java.io.IOException: Cannot run program “mapper.rb”: CreateProcess error=2, ?????????
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.io.IOException: CreateProcess error=2, ?????????
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.(ProcessImpl.java:81)
    at java.lang.ProcessImpl.start(ProcessImpl.java:30)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
    … 20 more
36. yucc Oct 21.09 / 6am

    I’m using hadoop 0.20.1
    And I’ve tried the python code you provided but there’s no luck.
    The same “Error in configuring object”, quite annoying!
37. Michael G. Noll Oct 25.09 / 1pm

    Hello yucc, you might want to take a look at Ben’s questions and how we could solve his problem.
38. Ryan Nov 04.09 / 9am

    All,

    Any idea how to ship a library (Python Imaging Library in my case) along with the jar? I was hoping my job would just work, but I get an ImportError when running within Hadoop.

    Thanks,
    Ryan
39. Ben Feb 05.10 / 7am

    Ryan, it dosnt work like that. You will need to install the PIL on each machine in the cluster to get it working.
40. didi Mar 13.10 / 1am

    Cheers exactly what I was looking for. Excellent tutorial.
41. Gary Apr 26.10 / 7pm

    I’m trying to get a python wordCount program running in a Cygwin environment. The Hadoop job is failing with
    “Streaming Job Failed!”. Any ideas?

    $ pwd
    /home/ditlow/hadoop-0.20.2

    $ cat go
    #bin/stop-all.sh
    #bin/hadoop namenode -format
    #bin/start-all.sh

    bin/hadoop fs -rmr output
    #bin/hadoop fs -rmr input
    #bin/hadoop fs -mkdir input
    #bin/hadoop fs -put text1 text2 input
    #bin/hadoop fs -ls input
    #bin/hadoop fs -cat input/text1
    #bin/hadoop fs -cat input/text2

    #echo $(cygpath -aw ./)

    bin/hadoop jar contrib/streaming/hadoop-0.20.2-streaming.jar \
    -file ./mapper.py \
    -mapper ./mapper.py \
    -file ./reducer.py \
    -reducer ./reducer.py \
    -input input/* \
    -output output \

    bin/hadoop fs -ls output
    bin/hadoop fs -cat output/_logs

    $ cat ./mapper.py
    #!/usr/bin/env python

    import sys

    for line in sys.stdin:
    line = line.strip()
    words = line.split()
    for word in words:
    print “%s %s” % (word,1)

    $ cat ./reducer.py
    #!/usr/bin/env python

    import sys

    h = {}
    for line in sys.stdin:
    line = line.strip()
    word,ct = line.split()
    h[word] = h.get(word,0) + int(ct)

    for word in h:
    print “%s %s” % (word,h[word])

    $ ./go
    Deleted hdfs://localhost:9000/user/ditlow/output
    packageJobJar: [./mapper.py, ./reducer.py, /C:/tmp/hadoop-ditlow/hadoop-unjar1381245209596335427/] [] C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\streamjob2933274506157772916.jar tmpDir=null
    10/04/26 11:45:21 INFO mapred.FileInputFormat: Total input paths to process : 2
    10/04/26 11:45:21 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-ditlow/mapred/local]
    10/04/26 11:45:21 INFO streaming.StreamJob: Running job: job_201004261135_0003
    10/04/26 11:45:21 INFO streaming.StreamJob: To kill this job, run:
    10/04/26 11:45:21 INFO streaming.StreamJob: C:\Cygwin\home\ditlow\hadoop-0.20.2\/bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201004261135_0003
    10/04/26 11:45:21 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201004261135_0003
    10/04/26 11:45:22 INFO streaming.StreamJob: map 0% reduce 0%
    10/04/26 11:46:12 INFO streaming.StreamJob: map 100% reduce 100%
    10/04/26 11:46:12 INFO streaming.StreamJob: To kill this job, run:
    10/04/26 11:46:12 INFO streaming.StreamJob: C:\Cygwin\home\ditlow\hadoop-0.20.2\/bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201004261135_0003
    10/04/26 11:46:12 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201004261135_0003
    10/04/26 11:46:12 ERROR streaming.StreamJob: Job not Successful!
    10/04/26 11:46:12 INFO streaming.StreamJob: killJob…
    Streaming Job Failed!
    cat: Source must be a file.
42. Michael G. Noll Apr 26.10 / 10pm

    Hello Gary, I don’t have any experience with running Hadoop via Cygwin on Windows, unfortunately.

    I guess that

    cat: Source must be a file.

    is the result of

    # file: go
    …
    bin/hadoop fs -cat output/_logs

    ?
43. anil May 04.10 / 11am

    suppose if want to access the configuration info in the map reduce programs how is it possible?
    i have multiple files as input i need to find out the file name of each record while processing.

    how get the file name python code?

Michael-Noll.com and all contents copyright © 2004-2010 by Michael G. Noll, unless otherwise noted.

Entries RSS Comments RSS