HomeworkLib
Question

How can I access tables from hadoop (e.g. hive) using PySpark in Python? I am using...

How can I access tables from hadoop (e.g. hive) using PySpark in Python? I am using a work laptop which has access to the company Hadoop Server.

engineering   Computer-Science   
0 0
Add a comment Improve this question Transcribed image text
Next > < Previous
Sort answers by oldest

Homework Answers

Answer #1

Please find the steps to connect hive to PySpark below :

  1. Copy the following files from Hive cluster to your /conf folder in PySpark :
    hive-site.xml
    hdfs-site.xml
    hbase-site.xml
    core-site.xml
  2. Copy all the 'jar' files that your project uses to PySpark's /jar folder.
  3. Enable Hive support in your PySpar from settings. You are now ready and can run PySpark paired with Hive !
  4. Run the following code to set up the pairing --
from pyspark.sql import SparkSession
from pyspark.sql import Rowspark = SparkSession \
    .builder \
    .appName("Name of your App") \
    .enableHiveSupport() \
    .getOrCreate()

5. Now you can query the Hive table directly from spark!

Ex code :  >> spark.sql("show tables").show()
0 0
Add a comment
Know the answer?
Add Answer to:
How can I access tables from hadoop (e.g. hive) using PySpark in Python? I am using...
Your Answer:

Post as a guest

Your Name:

What's your source?

Earn Coins

Coins can be redeemed for fabulous gifts.

Log In

Sign Up

Not the answer you're looking for? Ask your own homework help question. Our experts will answer your question WITHIN MINUTES for Free.
Similar Homework Help Questions

  • I am new at prgraming in python and I was wondering how you can create a...

    I am new at prgraming in python and I was wondering how you can create a new line within a print statment. I tried using \n but i keep getting an error

  • Hello, I'm working on inserting data into a table using MySQL and Python on my Windows...

    Hello, I'm working on inserting data into a table using MySQL and Python on my Windows OS laptop; and I am asked to modify the script so that it executes an insert query in one of my database tables. I am also asked to print the table before and after I execute this query in order to ensure the new information was inserted into the table. I'd like to INSERT INTO my EMPLOYEE table. The columns are employee_id, employee_password, order_id...

  • Python Question. I am trying to connect to a database and create a table using Python...

    Python Question. I am trying to connect to a database and create a table using Python in this code but I keep getting an error on line 48 saying that "Quotes" is not defined. Could anyone please help me set this up correctly? I am using Azure SQL Database to create this. import pyodbc import json import requests as r import pprint connection_string = 'Driver={ODBC Driver 17 for SQL Server};' \ 'Server=randomquotegenerator.database.windows.net,1433;' \ 'Database=RandomQuoteGenerator;' \ 'Uid=carterholliday;' \ 'Pwd=MIS54002k19;' \ 'TrustServerCertificate=no;'...

  • Hi I need some help writing a security code using python mongodb restful api. I just...

    Hi I need some help writing a security code using python mongodb restful api. I just need help on how to provide a security login to user to enter their username and password or to have a block in using extra access once they have logined and they want more access to the databases they will be force to sign out of the server but I just do not know how to start on it can someone show me how...

  • I am using python and I want to know how I can get the spacing between...

    I am using python and I want to know how I can get the spacing between the low mid and high to be dependent on the range and how I can get that boarder to be dependent on the range as well. Im not sure what you mean, I just want to know about the spacing. Modify your project1.py Python program so that it displays the low, mid and high point values for periodic function and a double-line border. (HINT:...

  • How can I use python to extract the headline, the news source (usually the name of...

    How can I use python to extract the headline, the news source (usually the name of a newspaper, magazine, TV or radio station), and the publication date/time, from an HTML document of a news story? Using python I have to make a program that will read an HTML document of a news story, how can I use python to find the above information? these are the modules I have to use; from re import findall, finditer, MULTILINE, DOTALL from tkinter.scrolledtext...

  • Explain how CIA can be achieved down to the folder and data file access level for departments and users using Active Dir...

    Explain how CIA can be achieved down to the folder and data file access level for departments and users using Active Directory and Windows Server 2012 R2 access control configurations. Configuring unique access controls for different user types is an example of which kind of access controls?

  • This is a technical setup question regarding server-side swift (Apple's programming language). I have currently got...

    This is a technical setup question regarding server-side swift (Apple's programming language). I have currently got a swift package with kitura dependency on a ubuntu 16.04 machine, which starts a server. (accessed through ssh on mac) I have followed this course from Udacity: server-side swift, and I have reached lesson 1 part 11 but I am now stuck. I am unable to open `hostname:8080` on my mac host, which is how I have setup the port in my main.swift using...

  • Using Eclipse I am trying to pass a string of numbers from the client class to...

    Using Eclipse I am trying to pass a string of numbers from the client class to the server. The server will then compute the sum, average max and min from the numbers and then pass it back to the client. The client will then display the answers in a text field. This is what i have so far for the class client and not sure how to proceed or if it's correct. i also am not sure how to complete...

  • Hi, I am having trouble with using the matplotlib function in python. A tutorial in this...

    Hi, I am having trouble with using the matplotlib function in python. A tutorial in this question and a lesson on how to use it would be great. Thanks so much. 2- Write a program using matplotlib that gives the following output Consider 100 points between [0,10] Fiqure 1 10 8 10 #Your program here:

ADVERTISEMENT
Free Homework Help App
Download From Google Play
Scan Your Homework
to Get Instant Free Answers
Need Online Homework Help?
Ask a Question
Get Answers For Free
Most questions answered within 3 hours.
ADVERTISEMENT
ADVERTISEMENT