Skip to main content leehalls.net

Pandas SQL and Pain

I’ve left the home monitoring system alone for a few years, its just sat quietly in the garage collating data into an SQL database, the only changes made recently were to move everything onto a Pi4 and add an additional camera into the motioneye system running on the same Pi. So as its a slow day i decided now was the time to update it or rather start work on it. One thing i wanted to do was re-instigate the live readings and an auto graph element and for this i needed to extract data - simple i got my old code somewhere …. somewhere … maybe.

After a bit of searching and playing around i decided to re-write;

Though I spent far longer testing this than i should have. The code when run will sometimes report correctly a full dataframe and yet next time with exactly the same code (no changes) it reports an empty one. Run it > empty > run it > empty > run it > data

Not sure why - yet.

Anyway ignoring why it sometimes works and then others dont for now the code is very simple and it outputs an interactive visualisation like;

You can find the interactive version here:

python code snippet start

import sqlite3
import pandas as pd
import pandas_bokeh
from sqlalchemy import create_engine
import datetime as dt
file = "home_mon_db.sqlite"
pandas_bokeh.output_file('data_out.html')

#define time window
timenow = dt.datetime.now()
# testing values
now_date = '2021-02-11'
#prev_time = '06:00:00'
#now_time = '08:00:00'
# programatic values
#now_date = timenow.strftime('%Y-%m-%d')
prev_time = str((dt.datetime.now() - dt.timedelta(hours=8,minutes=0)).strftime('%H:%M:%S')) # defines a window to get data from
now_time = str(timenow.strftime('%H:%M:%S'))

print('DATE:',now_date,'PREVTIME:',prev_time,'NOWTIME',now_time)

try:
        cnx = sqlite3.connect('home_mon_db.sqlite')
        df = pd.read_sql("SELECT * FROM home_mon WHERE DATE = ? AND time BETWEEN ? AND ?", cnx, params={now_date, prev_time, now_time})
except ValueError as e:
  print('failed', e)

if df.empty == True:
        print('DataFrame is empty!')
else:
        print(df)
        sel =df[['TIME','EXTERNAL']]
        # sel.plot_bokeh(rangetool=True)  # this i like
        sel.plot_bokeh(
                kind='line',
                x='TIME',
                xlabel='TIME',
                ylabel='Degrees C',
                title='Sensor inside garage'
        )

print('finished')
cnx.close()

python code snippet end