Last active
April 11, 2023 06:03
-
-
Save ThomasMiconi/5f164404f9cee7fca22662ef35bb2ae4 to your computer and use it in GitHub Desktop.
Prompt for a novel arithmetic operation, following Zhou et al. 2022. See comments.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: 821`763= | |
Explanation: | |
The first number is 821, FN=[8,2,1]. The second number is 763, SN=[7,6,3]. Since FN [8,2,1] has 3 digits, SN [7,6,3] has 3 digits, thus the maximum number of digits is 3. In each subsequent step, we remove one number from the beginning of FN and one from the beginning of SN. | |
Length of FN is 3. FN=[8,2,1]. Length of SN is 3. SN=[7,6,3]. FN[3]=8. SN[3]=7. C[3]=0. Since 8*7=56, 56%10=6, 56//10=5. Length of A is 1. Thus A=[6]. Since 5 is odd, we change the value of C. Since C[3]=0, C[2]=1. | |
Length of FN is 2. FN=[2,1]. Length of SN is 2. SN=[6,3]. FN[2]=2. SN[2]=6. C[2]=1. Since 2*6=12, 12%10=2, 12//10=1. Length of A is 2. Thus A=[6,2]. Since 1 is odd, we change the value of C. Since C[2]=1, C[1]=0. | |
Length of FN is 1. FN=[1]. Length of SN is 1. SN=[3]. FN[1]=1. SN[1]=3. C[1]=0. Since 1*3=4, 4%10=4, 4//10=0. Length of A is 3. Thus A=[6,2,4]. Since 0 is even, we do not change the value of C. Since C[1]=0, C[0]=0. | |
There are no more digits and C[0]=0. Thus the process is complete. Since there are no more operators, the problem is complete. The final Answer is [6,2,4]. | |
Problem: 9098`291= | |
Explanation: | |
The first number is 9098, FN=[9,0,9,8]. The second number is 291, SN=[2,9,1]. Since FN [9,0,9,8] has 4 digits, SN [2,9,1] has 3 digits, thus the maximum number of digits is 4. In each subsequent step, we remove one number from the beginning of FN and one from the beginning of SN. | |
Length of FN is 4. FN=[9,0,9,8]. Length of SN is 3. SN=[2,9,1]. FN[4]=9. SN[4]=2. C[4]=0. Since 9*2=18, 18%10=8, 18//10=1. Length of A is 1. Thus A=[8]. Since 1 is odd, we change the value of C. Since C[4]=0, C[3]=1. | |
Length of FN is 3. FN=[0,9,8]. Length of SN is 2. SN=[9,1]. FN[3]=0. SN[3]=9. C[3]=1. Since 9*0=0, 0%10=0, 0//10=0. Length of A is 2. Thus A=[8,0]. Since 0 is even, we do not change the value of C. Since C[3]=1, C[2]=1. | |
Length of FN is 2. FN=[9,8]. Length of SN is 1. SN=[1]. FN[2]=9. SN[2]=1. C[2]=1. Since 9*1=9, 9%10=9, 9//10=0. Length of A is 3. Thus A=[8,0,9]. Since 0 is even, we do not change the value of C. Since C[2]=1, C[1]=1. | |
Length of FN is 1. FN=[8]. Length of SN is 0. SN=[]. FN[1]=8. SN is empty. C[1]=1. Since 8*0=0, 0%10=0, 0//10=0. Length of A is 4. Thus A=[8,0,9,0]. Since 0 is even, we do not change the value of C. Since C[1]=1, C[0]=1. | |
There are no more digits but C[0]=1. Length of A is 5. Thus A=[8,0,9,0,1]. | |
There are no more digits and the process is complete. Since there are no more operators, the problem is complete. The final Answer is [8,0,9,0,1]. | |
Problem: 743`128= | |
Explanation: | |
The first number is 743, FN=[7,4,3]. The second number is 128, SN=[1,2,8]. Since FN [7,4,3] has 3 digits, SN [1,2,8] has 3 digits, thus the maximum number of digits is 3. In each subsequent step, we remove one number from the beginning of FN and one from the beginning of SN. | |
Length of FN is 3. FN=[7,4,3]. Length of SN is 3. SN=[1,2,8]. FN[3]=7. SN[3]=1. C[3]=0. Since 7*1=7, 7%10=7, 7//10=0. Length of A is 1. Thus A=[7]. Since 0 is even, we do not change the value of C. Since C[3]=0, C[2]=0. | |
Length of FN is 2. FN=[4,3]. Length of SN is 2. SN=[2,8]. FN[2]=4. SN[2]=2. C[2]=0. Since 4*2=8, 8%10=8, 8//10=0. Length of A is 2. Thus A=[7,8]. Since 0 is even, we do not change the value of C. Since C[2]=0, C[1]=0. | |
Length of FN is 1. FN=[3]. Length of SN is 1. SN=[8]. FN[1]=3. SN[1]=8. C[1]=0. Since 8*3=24, 24%10=4, 24//10=2. Length of A is 3. Thus A=[7,8,4]. Since 2 is even, we do not change the value of C. Since C[1]=0, C[0]=0. | |
There are no more digits and C[0]=0. Thus the process is complete. Since there are no more operators, the problem is complete. The final Answer is [7,8,4]. | |
Problem: 862`7645= | |
Explanation: | |
The first number is 862, FN=[8,6,3]. The second number is 7645, SN=[7,6,4,5]. Since FN=[8,6,3] has 3 digits, SN=[7,6,4,5] has 4 digits, thus the maximum number of digits is 4. In each subsequent step, we remove one number from the beginning of FN and one from the beginning of SN. | |
Length of FN is 3. FN=[8,6,3]. Length of SN is 4. SN=[7,6,4,5]. FN[4]=8. SN[4]=7. C[4]=0. Since 8*7=56, 56%10=6, 56//10=5. Length of A is 1. Thus A=[6]. Since 5 is odd, we change the value of C. Since C[4]=0, C[3]=1. | |
Length of FN is 2. FN=[6,3]. Length of SN is 3. SN=[6,4,5]. FN[3]=6. SN[3]=6. C[3]=1. Since 6*6=36, 36%10=6, 36//10=3. Length of A is 2. Thus A=[6,6]. Since 3 is odd, we change the value of C. Since C[3]=1, C[2]=0. | |
Length of FN is 1. FN=[3]. Length of SN is 2. SN=[4,5]. FN[2]=3. SN[2]=4. C[2]=0. Since 3*4=12, 12%10=2, 12//10=1. Length of A is 3. Thus A=[6,6,2]. Since 1 is odd, we change the value of C. Since C[2]=0, C[1]=1. | |
Length of FN is 0. FN=[]. Length of SN is 1. SN=[5]. FN is empty. SN[1]=5. C[1]=1. Since 0*5=0, 0%10=0, 0//10=0. Length of A is 4. Thus A=[6,6,2,0]. Since 0 is even, we do not change the value of C. Since C[1]=1, C[0]=1. | |
There are no more digits but C[0]=1. Length of A is 5. Thus A=[6,6,2,0,1]. | |
There are no more digits and the process is complete. Since there are no more operators, the problem is complete. The final Answer is [6,6,2,0,1]. | |
Problem: 17203`891457= |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is essentially an additional control for Teaching Algorithmic Reasoning via In-context Learning by Hattie Zhou (@hlml) et al.
This paper showed that large language models (LLMs) can learn to perform multi-step arithmetic operations, in-context and in a way that generalizes to arbitrary-size numbers, simply by explaining them carefully how to do it.
One caveat is that this paper used the common arithmetic operations (addition, subtraction, multiplication). Because there are many web pages explaining how to perform these operations, this left open the possibility that these results were at least in part influenced by some form of contamination.
This prompt describes a totally novel (and deliberately bizarre) arithmetic operation: pointwise multiplication modulo 10 with binary evenness-based carry, left-to-right Given two numbers, the result is computed as follows:
Like other arithmetic operations, performing this operation on arbitrary numbers requires learning a complex, open-ended, stateful algorithm. Unlike other arithmetic operations, it cannot possibly be subject to contamination, since this operation is entirely novel and thus hasn't been described before.
Yet, based on this prompt alone, LLMs can learn to perform this novel operation, including on numbers larger than anything seen in the prompt. This confirms the paper's original results: LLMs do not merely apply stored patterns from their training, but can identify novel complex (algorithmic) patterns in their input, and then generalize them to novel, unseen inputs.
Implications concerning "LLM intelligence" are left to the reader.
The prompt is adapted from Zhou et al.'s original addition prompt. Confirmed to work on GPT 3.5 (complete-mode) with text-davinci-003 engine.