Skip to content

Instantly share code, notes, and snippets.

@FurcyPin

FurcyPin/bigquery-frame_analyze_example-part2.py

Last active February 20, 2022 10:39
Show Gist options
  • Save FurcyPin/f27ebc184fc37266ec52e78251a56649 to your computer and use it in GitHub Desktop.
Save FurcyPin/f27ebc184fc37266ec52e78251a56649 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
bigquery-frame_analyze_example-part2.py
df = bigquery.sql(query)
df.show()
# +----+------------+---------------------+--------------------------------------------+
# | id | name | types | evolution |
# +----+------------+---------------------+--------------------------------------------+
# | 1 | Bulbasaur | ['Grass', 'Poison'] | {'can_evolve': True, 'evolves_from': None} |
# | 2 | Ivysaur | ['Grass', 'Poison'] | {'can_evolve': True, 'evolves_from': 1} |
# | 3 | Venusaur | ['Grass', 'Poison'] | {'can_evolve': False, 'evolves_from': 2} |
# | 4 | Charmander | ['Fire'] | {'can_evolve': True, 'evolves_from': None} |
# | 5 | Charmeleon | ['Fire'] | {'can_evolve': True, 'evolves_from': 4} |
# | 6 | Charizard | ['Fire', 'Flying'] | {'can_evolve': False, 'evolves_from': 5} |
# | 7 | Squirtle | ['Water'] | {'can_evolve': True, 'evolves_from': None} |
# | 8 | Wartortle | ['Water'] | {'can_evolve': True, 'evolves_from': 7} |
# | 9 | Blastoise | ['Water'] | {'can_evolve': False, 'evolves_from': 8} |
# +----+------------+---------------------+--------------------------------------------+
analyzed_df = analyze(df)
# Analyzing 5 columns ...
analyzed_df\
.withColumn("most_frequent_value", f.expr("approx_top_100[OFFSET(0)]"))\
.drop("approx_top_100").show()
# +------------------------+-------------+-------+----------------+------------+-----------+-----------+------------------------------------+
# | column_name | column_type | count | count_distinct | count_null | min | max | most_frequent_value |
# +------------------------+-------------+-------+----------------+------------+-----------+-----------+------------------------------------+
# | id | INTEGER | 9 | 9 | 0 | 1 | 9 | {'value': '1', 'count': 1} |
# | name | STRING | 9 | 9 | 0 | Blastoise | Wartortle | {'value': 'Bulbasaur', 'count': 1} |
# | types! | STRING | 13 | 5 | 0 | Fire | Water | {'value': 'Grass', 'count': 3} |
# | evolution.can_evolve | BOOLEAN | 9 | 2 | 0 | false | true | {'value': 'true', 'count': 6} |
# | evolution.evolves_from | INTEGER | 9 | 6 | 3 | 1 | 8 | {'value': 'NULL', 'count': 3} |
# +------------------------+-------------+-------+----------------+------------+-----------+-----------+------------------------------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment