An occasionally updated blog, mostly related to programming. (Views are my own, not my employer's.)
Wednesday, October 11, 2023
Meow
Sunday, July 16, 2023
Boston Housing dataset ethical/data quality problems
ImportError:
`load_boston` has been removed from scikit-learn since version 1.2.
The Boston housing prices dataset has an ethical problem: as
investigated in [1], the authors of this dataset engineered a
non-invertible variable "B" assuming that racial self-segregation had a
positive impact on house prices [2]. Furthermore the goal of the
research that led to the creation of this dataset was to study the
impact of air quality but it did not give adequate demonstration of the
validity of this assumption.
The scikit-learn maintainers therefore strongly discourage the use of
this dataset unless the purpose of the code is to study and educate
about ethical issues in data science and machine learning.
In this special case, you can fetch the dataset from the original
source::
import pandas as pd
import numpy as np
data_url = "http://lib.stat.cmu.edu/datasets/boston"
raw_df = pd.read_csv(data_url, sep="\s+", skiprows=22, header=None)
data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
target = raw_df.values[1::2, 2]
Alternative datasets include the California housing dataset and the
Ames housing dataset. You can load the datasets as follows::
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
for the California housing dataset and::
from sklearn.datasets import fetch_openml
housing = fetch_openml(name="house_prices", as_frame=True)
for the Ames housing dataset.
[1] M Carlisle.
"Racist data destruction?"
[2] Harrison Jr, David, and Daniel L. Rubinfeld.
"Hedonic housing prices and the demand for clean air."
Journal of environmental economics and management 5.1 (1978): 81-102.
It was interesting reading M Carlisle's investigation and the scikit-learn discussion around it. I also found Fairlearn's discussion enlightening. (I
One other surprise for me in this is the existence of an Ames, Iowa, housing dataset. Article describing it: Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project [PDF] (2011) by the original creator of the dataset, Dean DeCock.
Saturday, June 3, 2023
Drawing with all the fonts available in Colab
import os
import unicodedata
from PIL import Image, ImageDraw, ImageFont
def get_fonts(default_size=20):
fonts = []
for root, dirs, files in os.walk('/usr/share/fonts/truetype/'):
for file in files:
if file.endswith('.ttf'):
fonts.append(ImageFont.truetype(os.path.join(root, file), default_size))
return fonts
TARGET_ALPHABET = [c for c in map(chr, range(128))
if not unicodedata.category(c).startswith('C')]
FONTS = get_fonts()
WIDTH = 500
out = Image.new("RGB", (WIDTH, 2000), (255, 255, 255))
d = ImageDraw.Draw(out)
x = 0
y = 0
for c in TARGET_ALPHABET:
for font in FONTS:
d.text((x, y), c, font=font, fill=(0, 0, 0))
left, top, right, bottom = d.textbbox((x, y), c, font=font)
if right < WIDTH - 20:
x = right + 5
else:
x = 0
y = bottom + 5
out.show()
UNICODE CHARACTER DATABASE: General_Category Values
ImageDraw.textbbox
Monday, May 29, 2023
Handwriting recognition links
- IAM Handwriting Database
- EMNIST Dataset
- Kaggle notebook from Rohan Kulkarni: EMnist data OCR
- Keras: Handwriting recognition (from 2016)
- Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks by Alex Graves and J&7uml;rgen Schmidhuber (2008)
- Optical character recognition via deep learning (student project for Stanford CS 230 2018)
Wednesday, May 3, 2023
Jokes from Sidney Morgenbesser
- On pragmatism: "It's all very well in theory, but it doesn't work in practice."
- J.L. Austin is saying that while there are double negatives to express a positive -- "she is not uninteresting" -- there are no double positives to express a negative. Morgenbesser interrupts: "Yeah, yeah".
- Eating dinner at a restaurant, the waitress says there are two choices for pie: apple or blueberry. Morgenbesser says he'll have the apple. The waitress comes back a minute later to say there's actually a third choice, cherry. Morgenbesser: "In that case, I'll have the blueberry pie."
Saturday, February 11, 2023
Impressed with sympy
import sympy
from sympy import Rational
a = sympy.symbols('a')
e = Rational(13, 4) * (5 - 6 * a) - Rational(1, 4) * (Rational(1, 3) * a - 3)
print(sympy.latex(e))
$$22 - \frac{307 a}{12}$$
Tuesday, August 30, 2022
The Bitter Lesson
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. Essential to these methods is that they can find good approximations, but the search for them should be by our methods, not by us. We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.
Are there any lessons for the social sciences here? E.g., should economists give up on trying to find general laws and instead create systems to find those patterns, perhaps in incomprehensible form? Of course, a machine found model doesn't need to be incomprehensible... even with an ordinary least squares regression model, the parameters are still "found by the machine". Also, Sutton is speaking to AI researchers so the "bitter lesson" may not be for everyone. Even if speech synthesis based on linguistic expertise wasn't competitive, linguistics is still useful, right?
