I tried adding the path to Pycharm as follows: The problem is: when I go to Pycharm and try to call pyspark, Pycharm can not found the module. However, I use Pycharm to write scripts in python. I would like start playing in order to learn more about MLlib. SparkContext available as sc, HiveContext available as sqlContext.
Pycharm no python interpreter driver#
using builtin-java classes where applicableġ6/01/08 14:46:47 INFO SecurityManager: Changing view acls to: userġ6/01/08 14:46:47 INFO SecurityManager: Changing modify acls to: userġ6/01/08 14:46:47 INFO SecurityManager: SecurityManager: authentication disabled ui acls disabled users with view permissions: Set(user) users with modify permissions: Set(user)ġ6/01/08 14:46:50 INFO Slf4jLogger: Slf4jLogger startedġ6/01/08 14:46:50 INFO Remoting: Starting remotingġ6/01/08 14:46:51 INFO Remoting: Remoting started listening on addresses 14:46:51 INFO Utils: Successfully started service 'sparkDrive r' on port 50199.ġ6/01/08 14:46:51 INFO SparkEnv: Registering MapOutputTrackerġ6/01/08 14:46:51 INFO SparkEnv: Registering BlockManagerMasterġ6/01/08 14:46:51 INFO DiskBlockManager: Created local directory at /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/blockmgr-769e6f91-f0e7-49f9-b45d-1b6382637c95ġ6/01/08 14:46:51 INFO MemoryStore: MemoryStore started with capacity 530.0 MBġ6/01/08 14:46:52 INFO HttpFileServer: HTTP File server directory is /private/var/folders/5x/k7n54drn1csc7w0j7vchjnmc0000gn/T/spark-8e4749ea-9ae7-4137-a0e1-52e410a8e4c5/httpd-1adcd424-c8e9-4e54-a45a-a735ade00393ġ6/01/08 14:46:52 INFO HttpServer: Starting HTTP Serverġ6/01/08 14:46:52 INFO Utils: Successfully started service 'HTTP file serve r' on port 50200.ġ6/01/08 14:46:52 INFO SparkEnv: Registering OutputCommitCoordinatorġ6/01/08 14:46:52 INFO Utils: Successfully started service 'SparkUI ' on port 4040.ġ6/01/08 14:46:52 INFO SparkUI: Started SparkUI at ġ6/01/08 14:46:53 WARN MetricsSystem: Using default name DAGScheduler for source because is not set.ġ6/01/08 14:46:53 INFO Executor: Starting executor ID driver on host localhostġ6/01/08 14:46:53 INFO Utils: Successfully started service '.netty.NettyBlockTransferService ' on port 50201.ġ6/01/08 14:46:53 INFO NettyBlockTransferService: Server created on 50201ġ6/01/08 14:46:53 INFO BlockManagerMaster: Trying to register BlockManagerġ6/01/08 14:46:53 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50201 with 530.0 MB RAM, BlockManagerId(driver, localhost, 50201)ġ6/01/08 14:46:53 INFO BlockManagerMaster: Registered BlockManager Using Spark 's default log4j profile: org/apache/spark/log4j-defaults.propertiesġ6/01/08 14:46:44 INFO SparkContext: Running Spark version 1.5.1ġ6/01/08 14:46:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform. Type "help", "copyright", "credits" or "license" for more information. I’m new with apache spark and apparently I installed apache-spark with homebrew in my macbook: Last login: Fri Jan 8 12: 52:04 on 2:~$ pyspark
0 kommentar(er)
