Animation of x ^ y % m for various m

Thanks to some example code and other generous help from Jeremy, I'm excited to have been able to create an animated GIF of x XOR y MOD m for various m in Autoplot.

import os.path

import jarray

from org.das2.graph import DasColorBar
from java import io
from java.awt import image as awtimage
from javax import imageio
import javax.imageio.metadata
import javax.imageio.stream

ARGB = imageio.ImageTypeSpecifier.createFromBufferedImageType(awtimage.BufferedImage.TYPE_INT_ARGB)

COMMON_GIF_ATTRIBUTES = {
    'disposalMethod': 'none',
    'userInputFlag': 'FALSE',
    'transparentColorFlag': 'FALSE',
    'transparentColorIndex': '0',
}

LOOP_NODE_ATTRIBUTES = {
  'applicationID': 'NETSCAPE',
  'authenticationCode': '2.0',
}

def set_node_attr(node, attributes_dict):
    for key, value in attributes_dict.items():
        node.setAttribute(key, value)

def write_gif(images, path, delay_tenths_sec=5, loop=True):
     if not images:
         return
     gif_writer = imageio.ImageIO.getImageWritersBySuffix('gif').next()
     write_param = gif_writer.getDefaultWriteParam()
     metadata = gif_writer.getDefaultImageMetadata(ARGB, write_param)
     metadata_format = metadata.getNativeMetadataFormatName()
     print metadata_format
     root = metadata.getAsTree(metadata_format)
     graphics_control_extension_node = imageio.metadata.IIOMetadataNode('GraphicControlExtension')
     root.appendChild(graphics_control_extension_node)
     graphics_control_extension_node.setAttribute('delayTime', '%d' % delay_tenths_sec)
     set_node_attr(graphics_control_extension_node, COMMON_GIF_ATTRIBUTES)
     if loop:
         app_extensions_node = imageio.metadata.IIOMetadataNode('ApplicationExtensions')
         root.appendChild(app_extensions_node)
         app_extension_node = imageio.metadata.IIOMetadataNode('ApplicationExtension')
         app_extensions_node.appendChild(app_extension_node)
         set_node_attr(app_extension_node, LOOP_NODE_ATTRIBUTES)
         app_extension_node.setUserObject(jarray.array([1, 0, 0], 'b'))
     metadata.setFromTree(metadata_format, root)
     out = imageio.stream.FileImageOutputStream(io.File(path))
     gif_writer.setOutput(out)
     gif_writer.prepareWriteSequence(None)
     for image in images:
         gif_writer.writeToSequence(imageio.IIOImage(image, None, metadata), write_param)
     gif_writer.endWriteSequence()
     out.close()
 

def plot_to_gif(plot_func, path, keyword_params=[], delay_tenths_sec=10, loop=True):
    images = []
    for params in keyword_params:
        plot_func(**params)
        images.append(writeToBufferedImage())
    write_gif(images, path, delay_tenths_sec=delay_tenths_sec, loop=loop)

def xor_mod_plot(n=256, modulus=9):
  x = outerProduct(linspace(0, n - 1, n), ones(n))
  plot(gt(bitwiseXor(x, transpose(x))  % modulus, zeros(n, n)), renderType='nnSpectrogram>rebin=noInterpolate')
  dom.plots[0].setColortable(DasColorBar.Type.GRAYSCALE)
  annotation(0, text='m = %d' % modulus, borderType='Rounded_Rectangle')

plot_to_gif(xor_mod_plot, os.path.expanduser('~/autoplot/xor_mod.gif'),
    keyword_params=[dict(modulus=m) for m in range(2, 255)],  delay_tenths_sec=50)

Chapter 4 of "The Mythical Man-Month"

Brooks: "I will contend that conceptual integrity is the most important consideration in system design." Brooks starts with a discussion of the (aesthetic?) superiority of the Reims cathedral, in which unity of design was followed, vs others where multiple architects left their mark over the centuries of work. To achieve this, Brooks suggests dividing work between architects and implementers or even using the surgical team organization discussed in the previous chapter. Thoughts:

• Reducing the number of people doing the architecture does not guarantee integrity. Even a single individual can (and will) be inconsistent if they are not careful, e.g., not being consistent in naming or ordering of input/output arguments. It seems what's needed instead are principles, including the one from Brooks here valuing conceptual integrity, and rules derived from those principles, consistently applied (ideally enforced automatically).
• The best, most harmonious set of such rules is something to be discovered. Involving more people gives more chances to discover better rules. It may be necessary to have leaders to render decisions and arrange for writing up the final product, but the flow of information is not one way.
• Brooks anticipates the objection that giving free rein to the architects may yield something impossible to implement well. It looks like he thinks part of the answer is just to hire more experienced architects... but how are they supposed to get that experience in his system? Would he count experience gained as a subordinate?
• Even if a project does end up with one architect, it seems they should be able to justify their decisions to the team, including implementers, with reference to broadly agreed upon principles and rules. The analogy with architecture makes me think of an artist who might resent having their decisions challenged. So to does the idea that integrity is achieved by having only one or two individuals make the design decisions. But we're not going for art. We're going for something that works well. (I'm imagining some parishioners sitting crushed in the rubble of a cathedral that collapsed because none of the builders dared point out a design flaw. "It hurts to have these stone blocks crushing my bones, but I really do appreciate how they're all styled the same way.")

Maybe I'm reading Brooks as advocating for a more top-down approach than he really intended. However, his discussion on "aristocracy" does seem to justify this interpretation. // Edit: I was evitably not being charitable enough. The next chapter starts with the following, emphasizing the importance of communication: "If one separates responsibility for functional specification from responsibility for building a cheap, fast product, what discipline bounds the architect's inventive enthusiasm? The fundamental answer is thoroughgoing, careful, and sympathetic communication between architect and builder."

x ^ y % 9

https://twitter.com/aemkei/status/1378106731386040322 shows a really interesting graphic generated using a simple rule: ((x ^ y) % 9) != 0. I was curious what it would take to reproduce this in Autoplot, a data browser/analysis system developed by my brother and others. Based on an example script from him, the following seems to work:

x = outerProduct(linspace(0, 255, 256), ones(256))
plot(gt(bitwiseXor(x, transpose(x))  % 9, zeros(256, 256)))

I'm pretty happy with how it turned out. I don't have time now to dig into what's going on with the image much now, but zooming in on one corner may help:

def xorModPlot(n=256, modulus=9):
  x = outerProduct(linspace(0, n - 1, n), ones(n))
  plot(gt(bitwiseXor(x, transpose(x))  % modulus, zeros(n, n)))

xorModPlot(n=32)

The 45 degree diagonal line makes sense since any integer XOR-ed with itself = 0. I was going to say that this raises another question, how much is accounted for the XOR operation alone. But, naturally, for any x != y, they differ in at least one bit and so x ^ y != 0 in that case. Notes:

• To get the script editor, remember to select "Script Panel" under the "Enable Feature" menu under "Options".
• You can find a download link for the Mac version of Autoplot on http://autoplot.org/latest/
• To get a black and white image, under the "style" panel, I selected Colortable: "grayscale" and Rebin: "noInterpolate".

Looking at Chapter 2 of "The Mythical Man-Month"

Trying to restate the main points of Chapter 2 of "The Mythical Man-Month" mostly in my own words:

The majority of software projects fail due to lack of calendar time.

Reasons:

1. Bad estimation techniques, exacerbated by not accounting for intermediate risks.
2. Estimates conflate effort with progress, not recognizing that calendar time cannot be traded for workers.
3. Lack of confidence in estimates leads managers to over promise.
4. Bad progress monitoring. Software engineering fails to incorporate best practices from other engineering disciplines.
5. When a project starts to fall behind, often new people are added, but this actually makes things worse.

Optimism

"All programmers are optimists". The flexibility of programming may contribute to this: "The programmer builds from pure thought-stuff: concepts and very flexible representations thereof." Having the power to do anything makes you think that you can do anything easily. However, then problems with our ideas creep in as bugs, making that optimism unrealistic.

(This great flexibility implies a big search space, and a need -- or at least possibliity -- for more creativity. Is it right that programmers are more prone to over optimism, accounting for the novelty of the problem being solved?)

In a big project with many subtasks, the probability that everything will go well is teeny. (Brooks says, "The probability that each will go well...", but I guess he meant, "The probability that all will go well...")

The SWE-Month

While cost necessarily increases with both the time taken and the number of workers, progress does not. "Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth."

The two are interchangeable only if work can be divided amongst workers without requiring communication between workers. "[I]t is not even approximately true for systems programming."

If a task is sequential by nature, it cannot be partitioned and so clearly cannot be sped up by adding more workers. Debugging can make tasks sequential. (Huh? Fixing one bug reveals another??)

For tasks that can be partitioned but require communication, the cost of communication must be accounted for.

Two types of communication costs:

1. Training: "[V]aries linearly with the number of workers" (Clearly part of it does -- the trainee's own time -- but since training materials can be reused, other parts are sublinear.)
2. Intercommunication: "If each part of a the task must be separately coordinated with each other part, the effort increases as n(n-1)/2."

The extra cost from communication may be so high that it actually ends up slowing the project down.

Systems test

(Most impacted by sequential constraints because it needs to happen after rest of system is complete? Can test driven development help here?)

Rule of thumb for planning software schedule:

• 1/3 planning
• 1/6 coding
• 1/4 component test, early system test
• 1/4 full system test

Although it's uncommon to budget half of project time for testing, in reality, the author observed testing to actually take this amount of time for most projects. Under budgeting for testing can be particularly disastrous because the bad news comes close to the promised delivery date, when the cost per day is at maximum and failures are most visible to customers.

Giving into customer/management pressure to overpromise: gutless estimating

Lacking confidence in their estimates, managers find it hard to push back on customer/management desire for unreasonably optimistic planned delivery dates. The solution is to improve those estimation methods but also to recognize that even a rough estimate is better than wishful thinking and so to push back.

Regenerative Schedule Disaster

Brooks's Law: Adding manpower to a late software project makes it later.

"How the Wall Street Journal visualized the 2020 election results"

How the Wall Street Journal visualized the 2020 election results

Tools mentioned:

• d3: "We use a lot of D3, it seems to be a pretty standard thing in our world."
• React: "Typically, I guess the landscape is very heavy with JavaScript nowadays, specifically with React as a front-end library. We’ve leveraged a lot of that with the work that we did here and it helped us make components that were easily put anywhere for us."
• Next.js: "I’ll also mention how important automation is for what we do. We were using a technology called Next.js to generate all of our election pages, and we were able to very easily create multiple modules based on this technology."

XOR

d <- data.frame(y=c(0, 1, 1, 0),
  x1=c(1, 1, 0, 0), x2=c(1, 0, 1, 0))
f <- glm(y~x1*x2, data=d, family=binomial(link="logit"))

> f$coefficients
(Intercept)          x1          x2       x1:x2 
  -23.56607    47.13214    47.13214   -94.26427
> f$fitted.values
           1            2            3            4 
5.826215e-11 1.000000e+00 1.000000e+00 5.826215e-11 
> f$fitted.values > 0.5
    1     2     3     4 
FALSE  TRUE  TRUE FALSE 

Binary representation of an int type value with bitset::to_string

Get the binary representation of an integral type using bitset's to_string method:

#include <bitset>
#include <iostream>
#include <string>

template <typename T>
std::string getBinary(T t) {
  unsigned long val = t;
  std::bitset<8 * sizeof t> bits(val);
  return bits.to_string();
}

int main() {
  using namespace std;
  while (cin) {
    unsigned short x;
    cin >> x;
    cout << getBinary(x) << '\n';
  }
}