GPT-3 thread on productivity hacks

[jabowery]

The Hutter Prize should replace the Turing test in the machine learning zeitgeist if not the popular mind. That it hasn’t done so is symptomatic of “The dog ate my homework!” mentality of the machine learning world specifically and the philosophy of natural science, epistemology and ontology generally.

As the guy who originally suggested the compression prize idea to Marcus back in 2005 there are a few pitfalls here that I’ve mentioned before but that bear repeating:

Algorithmic Information Theory is the general field of study arising from Kolmogorov Complexity, upon which Solomonoff Induction, Algorithmic Probability Theory and Minimum Description Length Principle are founded. Indeed, the later 3 are practically synonymous. Algorithmic Information Theory is probably the keyphrase entry point for people. Its essence can be distilled down to the idea that a “bit” of information must be considered a bit in a machine language program: The smallest possible executable archive of a dataset of observations. The shortest such program has a length in bits. That length is the Kolmogorov Complexity of the data and the program itself comprises the data’s Algorithmic Information. Discovering the Algorithmic Information of a set of observations is Solomonoff Induction.

Hutter’s AIXI AGI Theory = Sequential Decision Theory ∘ Algorithmic Information Theory
or
AGI = engineering ∘natural science
or
AGI = ought ∘ is

The process of discovering Algorithmic Information is Solomonoff Induction and may be considered the essence of data-driven natural science. This process is subject to the Halting problem: It is provably unprovable that one has found the smallest of all possible executable archives of a dataset. This is why people say Solomonoff Induction isn’t computable and this is the origin of the first layer of “The dog ate my homework!” out of brats parading around with literally tens if not hundreds of billions of dollars per year in the guise of scientists and machine learning experts making civilization-level decisions based on “The Science” as they put into practice their pet models. The laconic question to them is simply this:

Is the comparison of two integers computable?

[jabowery]

Yannic’s Galactica review starts with a great rant about a language model that is a step in the right direction (including citation generation that is a lot more useful that GPT-3’s).

Here’s what I wrote in response to Yannic’s rant about the Gutenberg Press vs Theocrats:

I’ve been, for some time (like since 1982, see “Videotex Networking and the American Pioneer” at my blog “Feral Observations”) saying we’re in a historic rhyme with the period after the invention of the Gutenberg Press. People who haven’t, by now, caught on to the relationship between centralized social policy and theocratic supremacy, are very likely acolytes of the modern theocratic supremacy. However, before launching into a modern Thirty Years War for social policy freedom from these modern loyalists, we should grant them a charity based on Algorithmic Information Theory as the most unbiased model selection criterion and enter into this “conversation”:

“If you loyalists insist on centralized imposition of social policy could you at least try to accept that the most UNbiased model selection criterion is the minimum size of any executable archive of the data? Yes, yes, we know that will require using lots of RAM and CPUs if not TPUs, but consider the cost of a modern rhyme with The Thirty Years War, which you will lose since you can’t be objective about reality – not to mention that Moore’s Law has been exponentially decreasing the cost of the aforementioned resources – OK? And Yes, yes, we know that ‘the data’ may, itself be ‘biased’ but then would you be happy if ‘the data’ included whatever data you use to operationally define what is ‘biased’ and what is not ‘biased’ from a scientific point of view (understanding this won’t necessarily accommodate the moral mandates of your theocracy)?”

Galactica is a step in the right direction because it incorporates quasi-recurrent algorithmic reasoning without the authors realizing that is what they are doing. Yannic goes into that regarding the way “external working memory” tries to explicate the steps in reasoning so that external algorithmic executions can be invoked after training during execution of the model.

[jabowery]

There is a paper on extracting “Truth” from large language models. Once again there is no grounding in the principle of Algorithmic Information so I bothered to post a response suggesting why they might consider, at least, doing parameter distillation as a pre-processing step.

[jabowery]

@jabowery

What they are really bragging about may be compared to the process of going from a gzip to bzip2: First you gunzip and then you bzip2. The intermediate stage is a higher dimensional “model” that permits you to then collapse down with a superior lossless compression algorithm. Another way to think about this is going from a pixel video stream to a voxel video stream and then collapse down to a geometric model of the environment through which the camera is traveling. The intermediate representation is far larger than the data source let alone the final.

An extreme exploration of this kind of overparameterized enhancement of ML is in the paper “It’s Hard For Neural Networks to Learn the Game of Life”. As everyone knows, the rules of the Game of Life are quite simple but result in systems of high apparent complexity. In this paper, they try to train neural nets in two ways: 1) with a parameter count approximately the size necessary to encode the rules, and 2) an over-parameterized neural net – a “large” model. They found the “large” model converged on the rules but the small one did not.

This kind of study is really important because it exposes phenomena that, were there more serious ML researchers, would be the focus of obsessive study, e.g. what is it about the learning algorithms that permits the overparameterized model to escape from “overfitting”? If I were Kurzweil, I’d be breathlessly searching for any of Google’s employees that were asking that question and transporting them to the upper echelons where they could deploy the huge pile of economic rent in more productive directions. But then, if I were Kurzweil, I’d long ago have gotten Google to put $10B behind the Hutter Prize for Lossless Compression of Human Knowledge.

[jabowery]

@johnwalker

Ever since I started fooling with GPT-2, I have been calling language model synthesis programs “bullshit generators”. Here is how I explained it in a comment here on 2022-06-15,

Speaking of bullshit generators:

“Lawfare” as replacement for warfare takes a grotesque turn with LLM scaling laws demanding more electrical generation of “civil society” to generate more arcane legalese with which to obfuscate and de-obfuscate the logic of contractual obligations. Outisde the courtroom gaming language will be a major investment in langslop pollution just as it is now – providing various connotations* sufficient “adoption” to become actual word senses that the “Reasonable Man” will of course adopt during adjudication.

One might hope that this whole thing about “language” will turn into some sort of black hole to consume itself somewhere else – preferably in the intergalactic spaces devoid even of dark matter. But, of course, this cannot be. Electricity requires energy and energy requires matter.

Kilgore Trout novel material? Perhaps, but if you trust Surfshark with your privacy after reading this you’re a fool.

Here’s what led up to the following “customer service” interaction with Surfshark regarding their 30-day money back guarantee:

In the same session that I subscribed I was given an option to purchase a discounted extended subscription. I took that option.

Then I received two separate emails, second of which was for a much larger amount.

Within a week I went to get my money back. This is what ensued:

Maisie Caldwell says:
I want you to know that your experience truly matters to us. According to our policy, refunds are limited to one per service, and since you have already received a refund in the past, we are unable to process an additional one.

Our 30-day money-back guarantee is designed to let you try Surfshark risk-free for your first purchase, and you can find all the details here: https://surfshark.com/terms-of-service…

We have these guidelines in place to keep our refund policy fair for everyone, and I hope you can understand our position. That said, if you’re experiencing any issues, we are more than happy to help you resolve them - please let us know how we can assist.

If you decide to cancel your subscription’s auto-renewal, you can do this easily through your account’s Payments section by clicking “Cancel Renewal”: https://my.surfshark.com/account/subscription/payments…

Me:
This is not legitimate because the refund request form permitted me to request a refund for only one of the two – the smaller charge.

Maisie Caldwell says:
Please refer to my previous message for our Terms of Service.

Me:
Surfshark’s 30-day moneyback guarantee is a fraud. Here’s how they defraud you:
…

Want me to complete this?

Maisie Caldwell says:
I am sorry, but this is not a fraud. The Terms of Refund state that you may request a refund anytime, but refunds are limited to one per service.

* “connotations” in this context just brought to mind something I did when I was about 13 years old which, if anyone other than my brother had witnessed it, would have thought me utterly insane: I (playfully) attacked my brother physically yelling demonically “CONNOTATIONS!!! I WANT MORE CONNOTATIONS!!!”