Mapreduce katsetused Azure platvormil - Redigeerimiste ajalugu

PriitJ: /* Katsetus: ühised huvid */

2016-12-14T17:30:44Z

Katsetus: ühised huvid

@@ 149. rida: / 149. rida: @@
 Testandmed: https://snap.stanford.edu/data/higgs-social_network.edgelist.gz
 <code>mapper.py</code>
 <pre>
 </pre>
@@ 159. rida: / 244. rida: @@
 <pre>
 </pre>

PriitJ: /* MapReduce Azure platvormil */

2016-12-14T14:13:49Z

MapReduce Azure platvormil

←Vanem redaktsioon		Redaktsioon: 14. detsember 2016, kell 16:13
143. rida:		143. rida:

	hdfs dfs -put ml-20m/ratings.csv /example/data/		hdfs dfs -put ml-20m/ratings.csv /example/data/


	=== Katsetus: ~~otsing~~ ===		=== Katsetus: ühised huvid ===

			Sotsiaalmeedia andmete pealt: kes jälgib keda twitteris. Leiame kahe suvalise kasutaja ühised huvid.

			Testandmed: https://snap.stanford.edu/data/higgs-social_network.edgelist.gz

	<code>mapper.py</code>		<code>mapper.py</code>

PriitJ: /* MapReduce Azure platvormil */

2016-12-14T12:14:47Z

MapReduce Azure platvormil

@@ 16. rida: / 16. rida: @@
 #!/usr/bin/env python
 import sys
@@ 26. rida: / 25. rida: @@
 def main(separator='\t'):
      data = read_input(sys.stdin)
      # Process each words returned from read_input
@@ 44. rida: / 42. rida: @@
 #!/usr/bin/env python
 from itertools import groupby
 from operator import itemgetter
@@ 57. rida: / 54. rida: @@
 def main(separator='\t'):
      data = read_mapper_output(sys.stdin, separator=separator)
      # Group words and counts into 'group'
@@ 96. rida: / 92. rida: @@
 <pre>
 </pre>
@@ 102. rida: / 116. rida: @@
 <pre>
 </pre>
 === Katsetus: otsing ===

PriitJ: Uus lehekülg: '== MapReduce Azure platvormil == Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-introduction "Getting started" artikkel] Katsetustes kasutame she...'

2016-12-14T11:11:24Z

Uus lehekülg: '== MapReduce Azure platvormil == Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-introduction "Getting started" artikkel] Katsetustes kasutame she...'

Uus lehekülg

== MapReduce Azure platvormil ==

Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-introduction "Getting started" artikkel]

Katsetustes kasutame shelli, Pythonit ja stream-I/O-d, mis ei eelda mingite pakettide ja arenduskeskkondade installimist.

Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-streaming-python Pythoni tutorial]

[https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/FileSystemShell.html hdfs käsud]

=== Katsetus: word count ===

<code>mapper.py</code>

<pre>
#!/usr/bin/env python

# Use the sys module
import sys

# 'file' in this case is STDIN
def read_input(file):
# Split each line into words
for line in file:
yield line.split()

def main(separator='\t'):
# Read the data using read_input
data = read_input(sys.stdin)
# Process each words returned from read_input
for words in data:
# Process each word
for word in words:
# Write to STDOUT
print('%s%s%d' % (word, separator, 1))

if __name__ == "__main__":
main()
</pre>

<code>reducer.py</code>

<pre>
#!/usr/bin/env python

# import modules
from itertools import groupby
from operator import itemgetter
import sys

# 'file' in this case is STDIN
def read_mapper_output(file, separator='\t'):
# Go through each line
for line in file:
# Strip out the separator character
yield line.rstrip().split(separator, 1)

def main(separator='\t'):
# Read the data using read_mapper_output
data = read_mapper_output(sys.stdin, separator=separator)
# Group words and counts into 'group'
# Since MapReduce is a distributed process, each word
# may have multiple counts. 'group' will have all counts
# which can be retrieved using the word as the key.
for current_word, group in groupby(data, itemgetter(0)):
try:
# For each word, pull the count(s) for the word
# from 'group' and create a total count
total_count = sum(int(count) for current_word, count in group)
# Write to stdout
print("%s%s%d" % (current_word, separator, total_count))
except ValueError:
# Count was not a number, so do nothing
pass

if __name__ == "__main__":
main()
</pre>

Mingid sisendfailid on juba olemas:

hdfs dfs -ls /example/data/

Tõmbame nende pealt käima, <code>/example</code> on kirjutatav storage

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar -files mapper.py,reducer.py -mapper mapper.py -reducer reducer.py -input wasbs:///example/data/gutenberg/davinci.txt -output wasbs:///example/wordcountout

Väljund

hdfs dfs -text /example/wordcountout/part-00000 | head -50

=== Katsetus: sort ===

<code>mapper.py</code>

<pre>

</pre>

<code>reducer.py</code>

<pre>

</pre>

=== Katsetus: otsing ===

<code>mapper.py</code>

<pre>

</pre>

<code>reducer.py</code>

<pre>

</pre>