A key component of generating text from modern language models (LM) is the selection and tuning of decoding algorithms. These algorithms determine how to generate text from the internal probability distribution genera...
详细信息
ISBN:
(纸本)9798400700507
A key component of generating text from modern language models (LM) is the selection and tuning of decoding algorithms. These algorithms determine how to generate text from the internal probability distribution generated by the LM. The process of choosing a decoding algorithm and tuning its hyperparameters takes significant time, manual effort, and computation, and it also requires extensive human evaluation. Therefore, the identity and hyperparameters of such decoding algorithms are considered to be extremely valuable to their owners. In this work, we show, for the first time, that an adversary with typical API access to an LM can steal the type and hyperparameters of its decoding algorithms at very low monetary costs. Our attack is effective against popular LMs used in text generation APIs, including GPT-2, GPT-3 and GPT-Neo. We demonstrate the feasibility of stealing such information with only a few dollars, e.g., $0.8, $1, $4, and $40 for the four versions of GPT-3.
Several prior works have shown that language models (LMs) can generate text containing harmful social biases and stereotypes. While decoding algorithms play a central role in determining properties of LM generated tex...
详细信息
ISBN:
(纸本)9798350396904
Several prior works have shown that language models (LMs) can generate text containing harmful social biases and stereotypes. While decoding algorithms play a central role in determining properties of LM generated text, their impact on the fairness of the generations has not been studied. We present a systematic analysis of the impact of decoding algorithms on LM fairness, and analyze the trade-off between fairness, diversity and quality. Our experiments with top-p, top-k and temperature decoding algorithms, in open-ended language generation, show that fairness across demographic groups changes significantly with change in decoding algorithm's hyper-parameters. Notably, decoding algorithms that output more diverse text also output more texts with negative sentiment and regard. We present several findings and provide recommendations on standardized reporting of decoding details in fairness evaluations and optimization of decoding algorithms for fairness alongside quality and diversity.
Low density parity check (LDPC) code is a kind of linear block code with excellent performance which is proposed by Dr. Gallager. It overcomes the shortcomings of other error correcting codes and it is the nearest cha...
详细信息
Low density parity check (LDPC) code is a kind of linear block code with excellent performance which is proposed by Dr. Gallager. It overcomes the shortcomings of other error correcting codes and it is the nearest channel code from Shannon limit at the present discover. The decoding algorithms of LDPC code can reduce the decoding delay by the parallel decoding structure, so this has caused great concern in many fields. In recent years more and more scholars begin to study LDPC code, this makes the coding and decoding algorithms of LDPC code improved continuously. LDPC code has been widely used in many fields such as optical fiber communication, digital watermarking, underwater communication and deep space communication because of its excellent performance. Non-binary LDPC code has better performance compared with binary LDPC code. In this paper, FFT-BP decoding algorithm of non-binary LDPC code is studied, and on this basis, we proposed an improved FFT-BP algorithm. The improved decoding algorithm can effectively reduce the average iteration number. Finally, we achieved encode and decode algorithm of non-binary LDPC code with FPGA.
The McEliece cryptographic system is one of the finalists of the second round of the NIST Post-Quantum Cryptography Standardization Process. As such, it will likely be used in many instances to ensure secure communica...
详细信息
The McEliece cryptographic system is one of the finalists of the second round of the NIST Post-Quantum Cryptography Standardization Process. As such, it will likely be used in many instances to ensure secure communications over the Internet when quantum computers become practical. The selection of the new system, which will replace the widely used RSA cryptographic system, should be announced by 2025. The McEliece cryptographic system is code-based, which means that it uses an error-correcting code as its basis, typically a Goppa code; it is the random insertion of errors in the original message that guarantees the security of the system. Presumably, only the legitimate user is capable of correcting those errors, thereby retrieving the message for which it is intended. As a result, an efficient decoding algorithm is required. In this work, we study two decoding algorithms for Goppa codes, namely, the Euclidean and Patterson, implement them in Magma, and compare their performances considering several code parameters. Lastly, a modification to Patterson's algorithm is proposed to handle certain cases where the Goppa polynomial is reducible.
The LDPC code, namely low density parity check code, is essentially a linear block code. Its decoding performance is closer to the Shannon limit than the Turbo code. The definition and description of LDPC codes are in...
详细信息
The LDPC code, namely low density parity check code, is essentially a linear block code. Its decoding performance is closer to the Shannon limit than the Turbo code. The definition and description of LDPC codes are introduced in this paper. LDPC coding method includes the Algorithm of LU decomposable encoding and the algorithm of partitioned iterated encoding. the algorithm of message passing, the algorithm of minimum-sum decoding and the algorithm of bit-flipping decoding are the common decoding algorithms of the LDPC codes. The concept and features of the encoding and decoding algorithms are explored in this paper to provide some references for the relative researchers.
We analyze a new group-testing scheme, termed semi-quantitative group testing, which may be viewed as a concatenation of an adder channel and a discrete quantizer. Our focus is on non-uniform quantizers with arbitrary...
详细信息
We analyze a new group-testing scheme, termed semi-quantitative group testing, which may be viewed as a concatenation of an adder channel and a discrete quantizer. Our focus is on non-uniform quantizers with arbitrary thresholds. For the most general semi-quantitative group-testing model, we define three new families of sequences capturing the constraints on the code design imposed by the choice of the thresholds. The sequences represent extensions and generalizations of Bh and certain types of super-increasing and lexicographically ordered sequences, and they lead to code structures amenable for efficient recursive decoding. We describe the decoding methods and provide an accompanying computational complexity and performance analysis.
The development decoding algorithms of two-dimensional cross strip anodes image readouts for applications in UV astronomy are described. We present results with Monte Carlo simulation by GEANT4 toolkit, the results sh...
详细信息
ISBN:
(纸本)9781628415650
The development decoding algorithms of two-dimensional cross strip anodes image readouts for applications in UV astronomy are described. We present results with Monte Carlo simulation by GEANT4 toolkit, the results show that when the cross strip anode period is 0.5mm and the electrode width is 0.4mm, the spatial resolution accuracy is sufficient to reach better than 5 mu m, the temporal resolution accuracy of the event detection can be as low as 100 ps. The influences of the cross strip detector parameters, such as the anode period, the width of anode fingers (electrode), the width of the charge footprint at the anode (determined by the distance and the field between the MCP and the anode), the gain of the MCP and equivalent noise charge (ENC) are also discussed. The development decoding algorithms and simulation results can be useful for the designing and performance improvement of future photon counting imaging detectors for UV Astronomy.
In this paper, the performance of state of the art decoding methods of polar codes, such as the SC (Successive Cancellation), BP (belief propagation), LP (Linear Programming), LSC(List Successive Cancellation) and ML(...
详细信息
ISBN:
(纸本)9781479989201
In this paper, the performance of state of the art decoding methods of polar codes, such as the SC (Successive Cancellation), BP (belief propagation), LP (Linear Programming), LSC(List Successive Cancellation) and ML(Maximum Likelihood) decoding, over different binary-input discrete memoryless channels (B-DMCs) are presented. Simulation results indicate that decoding algorithms of polar codes have the performances of the estimated bit error rate (BER) below the order of 10-5. In addition, we came to the conclusion that BP algorithm outperforms SC at the cost of computational complexity. The performance of LP decoder is better than BP and the complexity is less than BP scheme, however, it can only be used in binary erasure channels (BECs). ML decoder has the best performance, but its high complexity makes it act as a reference to reveal the gap between ML and other algorithms. In order to remedy the performance deficiencies without any significant increase in decoding complexity, we further study the concatenated polar codes-the polar-LDPC(Low Density Parity Check) concatenation scheme, which is substantially outperforms ML decoding. What's more, the concatenated Polar-LDPC codes can remedy the error floor of LDPC codes. We firmly believed that the concatenated scheme of polar-LDPC would be a prominent technique in 5G (5th-generation) to support the more reliable transmission demand.
Quantum low-density parity-check (QLDPC) codes are promising candidates for error correction in quantum computers. One of the major challenges in implementing QLDPC codes in quantum computers is the lack of a universa...
详细信息
Quantum low-density parity-check (QLDPC) codes are promising candidates for error correction in quantum computers. One of the major challenges in implementing QLDPC codes in quantum computers is the lack of a universal decoder. In this work, we first propose to decode QLDPC codes with a belief propagation (BP) decoder operating on overcomplete check matrices. Then, we extend the neural BP (NBP) decoder, which was originally studied for suboptimal binary BP decoding of QLPDC codes, to quaternary BP decoders. Numerical simulation results demonstrate that both approaches as well as their combination yield a low-latency, high-performance decoder for several short to moderate length QLDPC codes.
The gradient descent bit-flipping with momentum (GDBF-w/M) and probabilistic GDBF-w/M (PGDBF-w/M) algorithms significantly improve the decoding performance of the bit-flipping (BF) algorithm. In this letter, we propos...
详细信息
The gradient descent bit-flipping with momentum (GDBF-w/M) and probabilistic GDBF-w/M (PGDBF-w/M) algorithms significantly improve the decoding performance of the bit-flipping (BF) algorithm. In this letter, we propose a channel-aware GDBF-w/M algorithm which operates deterministically based on the received values from the additive white Gaussian noise (AWGN) channel. Numerical results show that the proposed algorithm does not only mitigate the error-floor phenomenon of the GDBF-w/M algorithm, but it also has better decoding performance than the PGDBF-w/M algorithm without the need for a random number generator. Furthermore, the complexity of the proposed algorithm is slightly higher than that of the GDBF-w/M algorithm.
暂无评论