说明

过程比较繁琐,主要内容是安装配置运行sqoop、hadoop。

过程

(1)下载编译sqoop
从这个<页面><下载链接>

编译需要用到ant,从这个<页面><下载链接>

进入sqoop目录,运行:ant jar

将用数分钟时间下载maven2的库,目录在/root/.ivy2/cache下面。GWF下总是下载不成功,因此也没办法生成sqoop-1.4.6.jar。可以在国外主机上build后形成的sqoop-1.4.6.jar文件拷贝到lib目录下,或点<分享链接>

从sqoop的运行脚本exec ${HADOOP_COMMON_HOME}/bin/hadoop org.apache.sqoop.Sqoop "$@"看,它是依赖hadoop的。

(2)hadoop安装
从这个<页面><下载链接>

解压即安装,查看版本:bin/hadoop version

Hadoop 2.7.3
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff
Compiled by root on 2016-08-18T01:41Z
Compiled with protoc 2.5.0
From source with checksum 2e4ce5f957ea4db193bce3734ff29ff4
This command was run using /usr/local/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar

(3)配置sqoop
现在我们暂时不打算使用hadoop的其它功能,比如hbase/zookeeper等,编辑bin/configure-sqoop文件,注释掉相关内容,否则需要sqoop程序会提示: Please set $HBASE_HOME to the root of your HBase installation.。解决这些之后,可以运行sqoop help了。

(4)初试sqoop
将mysql的jar包拷贝到lib目录下,运行以下命令即表示已经连接上mysql:

#sqoop-1.4.6/bin/sqoop list-tables --connect jdbc:mysql://localhost:3306/test --username root -P
16/11/21 00:47:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Enter password:
16/11/21 00:47:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
user

(5)MYSQL与HDFS互导
尝试启动hadoop:sbin/start-df.sh,失败了:localhost: ssh: connect to host localhost port 22: Connection refused

安装ssh:apt-get install openssh-server

启动hadoop,如果提示:Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured,则编辑:vim etc/hadoop/core-site.xml

增加如下配置:

<property>
<name>fs.default.name</name>
<value>hdfs://192.168.142.129:9000</value>
</property>

重启hadoop。运行jps查看java进程,应该有如下进程:

14778 DataNode
10050 ResourceManager
14972 SecondaryNameNode
14620 NameNode
13528 NodeManager

如果localhost:50070无法访问,可能是datanode未启动,或者namenode未格式化,格式化namenode并重启:bin/hdfs namenode -format

导入数据到HDFS:

bin/sqoop import --connect jdbc:mysql://localhost:3306/test --driver com.mysql.jdbc.Driver --username root --password xxx --table user -m 1

成功了:

16/11/21 23:28:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/11/21 23:28:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/11/21 23:28:23 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
16/11/21 23:28:23 INFO manager.SqlManager: Using default fetchSize of 1000
16/11/21 23:28:23 INFO tool.CodeGenTool: Beginning code generation
16/11/21 23:28:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/21 23:28:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/21 23:28:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/tyc/dev/hadoop-2.7.3
Note: /tmp/sqoop-tyc/compile/82b7ff806df438edbf210a632c60e282/user.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/11/21 23:28:25 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-tyc/compile/82b7ff806df438edbf210a632c60e282/user.jar
16/11/21 23:28:25 INFO mapreduce.ImportJobBase: Beginning import of user
16/11/21 23:28:27 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/11/21 23:28:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/21 23:28:27 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/11/21 23:28:27 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/11/21 23:28:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/11/21 23:28:27 INFO db.DBInputFormat: Using read commited transaction isolation
16/11/21 23:28:27 INFO mapreduce.JobSubmitter: number of splits:1
16/11/21 23:28:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local373274648_0001
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-tyc/mapred/local/1479742108095/mysql-connector-java-5.1.9.jar <- /home/tyc/dev/sqoop-1.4.6/mysql-connector-java-5.1.9.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/mysql-connector-java-5.1.9.jar as file:/tmp/hadoop-tyc/mapred/local/1479742108095/mysql-connector-java-5.1.9.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-tyc/mapred/local/1479742108096/sqoop-1.4.6.jar <- /home/tyc/dev/sqoop-1.4.6/sqoop-1.4.6.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/sqoop-1.4.6.jar as file:/tmp/hadoop-tyc/mapred/local/1479742108096/sqoop-1.4.6.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-tyc/mapred/local/1479742108097/ant-eclipse-1.0-jvm1.2.jar <- /home/tyc/dev/sqoop-1.4.6/ant-eclipse-1.0-jvm1.2.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ant-eclipse-1.0-jvm1.2.jar as file:/tmp/hadoop-tyc/mapred/local/1479742108097/ant-eclipse-1.0-jvm1.2.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-tyc/mapred/local/1479742108098/ant-contrib-1.0b3.jar <- /home/tyc/dev/sqoop-1.4.6/ant-contrib-1.0b3.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ant-contrib-1.0b3.jar as file:/tmp/hadoop-tyc/mapred/local/1479742108098/ant-contrib-1.0b3.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-tyc/mapred/local/1479742108099/ivy-2.3.0.jar <- /home/tyc/dev/sqoop-1.4.6/ivy-2.3.0.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ivy-2.3.0.jar as file:/tmp/hadoop-tyc/mapred/local/1479742108099/ivy-2.3.0.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479742108095/mysql-connector-java-5.1.9.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479742108096/sqoop-1.4.6.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479742108097/ant-eclipse-1.0-jvm1.2.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479742108098/ant-contrib-1.0b3.jar
16/11/21 23:28:28 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479742108099/ivy-2.3.0.jar
16/11/21 23:28:28 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/11/21 23:28:28 INFO mapreduce.Job: Running job: job_local373274648_0001
16/11/21 23:28:28 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/11/21 23:28:28 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/11/21 23:28:28 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/11/21 23:28:28 INFO mapred.LocalJobRunner: Waiting for map tasks
16/11/21 23:28:28 INFO mapred.LocalJobRunner: Starting task: attempt_local373274648_0001_m_000000_0
16/11/21 23:28:28 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/11/21 23:28:28 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/11/21 23:28:28 INFO db.DBInputFormat: Using read commited transaction isolation
16/11/21 23:28:28 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
16/11/21 23:28:28 INFO db.DBRecordReader: Working on split: 1=1 AND 1=1
16/11/21 23:28:28 INFO db.DBRecordReader: Executing query: SELECT uid, name FROM user AS user WHERE ( 1=1 ) AND ( 1=1 )
16/11/21 23:28:28 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
16/11/21 23:28:28 INFO mapred.LocalJobRunner:
16/11/21 23:28:28 INFO mapred.Task: Task:attempt_local373274648_0001_m_000000_0 is done. And is in the process of committing
16/11/21 23:28:28 INFO mapred.LocalJobRunner:
16/11/21 23:28:28 INFO mapred.Task: Task attempt_local373274648_0001_m_000000_0 is allowed to commit now
16/11/21 23:28:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local373274648_0001_m_000000_0' to file:/home/tyc/dev/sqoop-1.4.6/user/_temporary/0/task_local373274648_0001_m_000000
16/11/21 23:28:28 INFO mapred.LocalJobRunner: map
16/11/21 23:28:28 INFO mapred.Task: Task 'attempt_local373274648_0001_m_000000_0' done.
16/11/21 23:28:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local373274648_0001_m_000000_0
16/11/21 23:28:28 INFO mapred.LocalJobRunner: map task executor complete.
16/11/21 23:28:29 INFO mapreduce.Job: Job job_local373274648_0001 running in uber mode : false
16/11/21 23:28:29 INFO mapreduce.Job:  map 100% reduce 0%
16/11/21 23:28:29 INFO mapreduce.Job: Job job_local373274648_0001 completed successfully
16/11/21 23:28:29 INFO mapreduce.Job: Counters: 15
    File System Counters
        FILE: Number of bytes read=3224839
        FILE: Number of bytes written=3536228
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=3
        Map output records=3
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=0
        Total committed heap usage (bytes)=47407104
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=29
16/11/21 23:28:29 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 2.229 seconds (0 bytes/sec)
16/11/21 23:28:29 INFO mapreduce.ImportJobBase: Retrieved 3 records.

再次运行,会提示…mapred FileAlreadyExistsException异常,删除文件夹: bin/hadoop fs -rmr /user/tyc/user

找到数据文件夹:bin/hdfs dfs -ls,找数据:bin/hdfs dfs -cat /user/tyc/user/part-m-00000,数据如下:

1,daniel
2,david

通过网页也能查看到一些信息:http://192.168.142.129:50070/explorer.html#/user/tyc/user

将HDFS的数据导出到mysql中:

bin/sqoop export --connect jdbc:mysql://localhost:3306/test2 --driver com.mysql.jdbc.Driver --username root --password dyzda30h --table user --export-dir hdfs://192.168.142.129:9000/user/tyc/user/part-m-00000

(192.168.142.129:9000是前面配置里的地址)

过程如下:

16/11/22 00:26:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/11/22 00:26:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/11/22 00:26:41 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
16/11/22 00:26:41 INFO manager.SqlManager: Using default fetchSize of 1000
16/11/22 00:26:41 INFO tool.CodeGenTool: Beginning code generation
16/11/22 00:26:41 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/22 00:26:41 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/22 00:26:41 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/tyc/dev/hadoop-2.7.3
Note: /tmp/sqoop-tyc/compile/6a2d4171caae60d833c1dd6a098f053d/user.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/11/22 00:26:43 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-tyc/compile/6a2d4171caae60d833c1dd6a098f053d/user.jar
16/11/22 00:26:43 INFO mapreduce.ExportJobBase: Beginning export of user
16/11/22 00:26:44 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/11/22 00:26:45 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user AS t WHERE 1=0
16/11/22 00:26:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/11/22 00:26:45 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
16/11/22 00:26:45 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/11/22 00:26:45 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/11/22 00:26:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/11/22 00:26:45 INFO input.FileInputFormat: Total input paths to process : 1
16/11/22 00:26:45 INFO input.FileInputFormat: Total input paths to process : 1
16/11/22 00:26:45 INFO mapreduce.JobSubmitter: number of splits:4
16/11/22 00:26:45 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
16/11/22 00:26:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local510358255_0001
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/mysql-connector-java-5.1.9.jar as file:/tmp/hadoop-tyc/mapred/local/1479745605859/mysql-connector-java-5.1.9.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/sqoop-1.4.6.jar as file:/tmp/hadoop-tyc/mapred/local/1479745605860/sqoop-1.4.6.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ant-eclipse-1.0-jvm1.2.jar as file:/tmp/hadoop-tyc/mapred/local/1479745605861/ant-eclipse-1.0-jvm1.2.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ant-contrib-1.0b3.jar as file:/tmp/hadoop-tyc/mapred/local/1479745605862/ant-contrib-1.0b3.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: Localized file:/home/tyc/dev/sqoop-1.4.6/lib/ivy-2.3.0.jar as file:/tmp/hadoop-tyc/mapred/local/1479745605863/ivy-2.3.0.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479745605859/mysql-connector-java-5.1.9.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479745605860/sqoop-1.4.6.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479745605861/ant-eclipse-1.0-jvm1.2.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479745605862/ant-contrib-1.0b3.jar
16/11/22 00:26:46 INFO mapred.LocalDistributedCacheManager: file:/tmp/hadoop-tyc/mapred/local/1479745605863/ivy-2.3.0.jar
16/11/22 00:26:46 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/11/22 00:26:46 INFO mapreduce.Job: Running job: job_local510358255_0001
16/11/22 00:26:46 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/11/22 00:26:46 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.sqoop.mapreduce.NullOutputCommitter
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Waiting for map tasks
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Starting task: attempt_local510358255_0001_m_000000_0
16/11/22 00:26:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/11/22 00:26:46 INFO mapred.MapTask: Processing split: Paths:/user/tyc/user/part-m-00000:12+2,/user/tyc/user/part-m-00000:14+3
16/11/22 00:26:46 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
16/11/22 00:26:46 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
16/11/22 00:26:46 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
16/11/22 00:26:46 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
16/11/22 00:26:46 INFO mapred.LocalJobRunner:
16/11/22 00:26:46 INFO mapred.Task: Task:attempt_local510358255_0001_m_000000_0 is done. And is in the process of committing
16/11/22 00:26:46 INFO mapred.LocalJobRunner: map
16/11/22 00:26:46 INFO mapred.Task: Task 'attempt_local510358255_0001_m_000000_0' done.
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local510358255_0001_m_000000_0
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Starting task: attempt_local510358255_0001_m_000001_0
16/11/22 00:26:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/11/22 00:26:46 INFO mapred.MapTask: Processing split: Paths:/user/tyc/user/part-m-00000:0+4
16/11/22 00:26:46 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
16/11/22 00:26:46 INFO mapred.LocalJobRunner:
16/11/22 00:26:46 INFO mapred.Task: Task:attempt_local510358255_0001_m_000001_0 is done. And is in the process of committing
16/11/22 00:26:46 INFO mapred.LocalJobRunner: map
16/11/22 00:26:46 INFO mapred.Task: Task 'attempt_local510358255_0001_m_000001_0' done.
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local510358255_0001_m_000001_0
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Starting task: attempt_local510358255_0001_m_000002_0
16/11/22 00:26:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/11/22 00:26:46 INFO mapred.MapTask: Processing split: Paths:/user/tyc/user/part-m-00000:4+4
16/11/22 00:26:46 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
16/11/22 00:26:46 INFO mapred.LocalJobRunner:
16/11/22 00:26:46 INFO mapred.Task: Task:attempt_local510358255_0001_m_000002_0 is done. And is in the process of committing
16/11/22 00:26:46 INFO mapred.LocalJobRunner: map
16/11/22 00:26:46 INFO mapred.Task: Task 'attempt_local510358255_0001_m_000002_0' done.
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local510358255_0001_m_000002_0
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Starting task: attempt_local510358255_0001_m_000003_0
16/11/22 00:26:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/11/22 00:26:46 INFO mapred.MapTask: Processing split: Paths:/user/tyc/user/part-m-00000:8+4
16/11/22 00:26:46 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
16/11/22 00:26:46 INFO mapred.LocalJobRunner:
16/11/22 00:26:46 INFO mapred.Task: Task:attempt_local510358255_0001_m_000003_0 is done. And is in the process of committing
16/11/22 00:26:46 INFO mapred.LocalJobRunner: map
16/11/22 00:26:46 INFO mapred.Task: Task 'attempt_local510358255_0001_m_000003_0' done.
16/11/22 00:26:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local510358255_0001_m_000003_0
16/11/22 00:26:46 INFO mapred.LocalJobRunner: map task executor complete.
16/11/22 00:26:47 INFO mapreduce.Job: Job job_local510358255_0001 running in uber mode : false
16/11/22 00:26:47 INFO mapreduce.Job:  map 100% reduce 0%
16/11/22 00:26:47 INFO mapreduce.Job: Job job_local510358255_0001 completed successfully
16/11/22 00:26:47 INFO mapreduce.Job: Counters: 20
    File System Counters
        FILE: Number of bytes read=12905440
        FILE: Number of bytes written=14145932
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=172
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=66
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Map-Reduce Framework
        Map input records=3
        Map output records=3
        Input split bytes=611
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=2
        Total committed heap usage (bytes)=224739328
    File Input Format Counters
        Bytes Read=0
    File Output Format Counters
        Bytes Written=0
16/11/22 00:26:47 INFO mapreduce.ExportJobBase: Transferred 172 bytes in 2.1283 seconds (80.8152 bytes/sec)
16/11/22 00:26:47 INFO mapreduce.ExportJobBase: Exported 3 records.