Information Theory MCQ || Information Theory Questions and Answers

Answer.2. S1 is less than S2

Explanation

Entropy of mutual information H(X):

$H\left( X \right) = \mathop \sum \nolimits_{i = 1}^n p\left( {{x_i}} \right).{\log _2}\frac{1}{{p\left( {{x_i}} \right)}} = {\log _2}m$

[for equiprobable ‘m’ no. of symbol]

Calculation:

Case-I: For source S₁;

No. of symbol; m = 4

H(x) = log₂ m

= 2 bits / symbol

Case-II: For source S₂;

No. of symbol; m = 16

H(x) = log₂ m

= 4 bits / symbol

∴ entropy of S₁ is less than S₂

Answer.1. 001

Explanation

The generator Matrix is given by

$G = \left[ {{I_K}{P^T}} \right]$

${P^T} = \left[ {\begin{array}{*{20}{c}} 1&1&0\\ 0&1&1\\ 1&1&1\\ 1&0&1 \end{array}} \right]$

The parity check matrix is given by:

H = [P I_{kn – K}]

Syndrome

S = eH^T

${H^T} = \left[ {\begin{array}{*{20}{c}} {{P^T}}\\ {{I_{n – k}}} \end{array}} \right]$ ${H^T} = \left[ {\begin{array}{*{20}{c}} 1&1&0\\ 0&1&1\\ 1&1&1\\ 1&0&1\\ 1&0&0\\ 0&1&0\\ 0&0&1 \end{array}} \right]$

S = eH^T

For error in 7^th Bit

E = [000 0001]

$S = \left[ {000\;000\;1} \right]\left[ {\begin{array}{*{20}{c}} 1&1&0\\ 0&1&1\\ 1&1&1\\ 1&0&1\\ 1&0&0\\ 0&1&0\\ 0&0&1 \end{array}} \right]$

S = [ 0 0 1]

Answer.4. 32 kilo bits / sec

Explanation

Shannon’s channel capacity is the maximum bits that can be transferred error-free. Mathematically, this is defined as:

$ C = B~{\log _2}\left( {1 + \frac{{\rm{S}}}{{\rm{N}}}} \right){\rm{bits}}$

B = Bandwidth of the channel

Signal to noise ratio = S/N

Calculation:

Given B = 4 kHz and SNR = 255

Channel capacity will be:

${\rm{C}} = 4~{\log _2}\left( {1 + 255} \right){\rm{kbits}}/{\rm{sec}}$

C = 32 kbits/sec

Answer.1. 0

Explanation

Mutual information of two random variables is a measure to tell how much one random variable tells about the other.

It is mathematically defined as:

I(X₁, X₂) = H(X₁) – H(X₁/X₂)

Application:

Since X₁ and X₂ are independent, we can write:

H(X₁/X₂) = H(X₁)

I(X₁,X₂ ) = H(X₁) – H(X₁)

= 0

Answer.2. 3

Explanation

Types of error correction

The codes which are used for both error detecting and error correction are called as “Error Correction Codes”. The error correction techniques are of two types. They are,

• Single error
• Burst error

Hamming code or Hamming Distance Code is the best error-correcting code we use in most communication network and digital systems.

Statement A is false.

Coding efficiency

The coding efficiency (η) of an encoding scheme is expressed as the ratio of the source entropy H(z) to the average codeword length L(z) and is given by

η = H(z)/L(z)

Since L(z) ≥ H(z) according to Shannon’s Coding theorem and both L(z) and H(z) are positive, 0≤ η ≤ 1

If m(z) is the minimum of the average codeword length obtained out of different uniquely decipherable coding schemes, then as per Shannon’s theorem, we can state that m(z) ≥ H(z)

Statement B is true.

Channel coding
When it comes to channel coding, we always have to compromise between energy efficiency and bandwidth efficiency.
Low-rate codes are not very efficient due to the reason that:

• They have large overheads
• Requires more bandwidth to transmit the data.
• The decoders are more complex.

While codes with greater redundancy are more efficient in correcting errors, as the efficiency of the code increases:

• It contributes greatly by operating at a lower transmit power and transmitting over long distances.
• It is also more tolerant towards interference and hence can be used with small antennas.
• Transmits at a higher data rate.

Hence Statement C is false while Statement D is true.

Error correction and detection

Any code which is capable of correcting the ‘t’ errors should satisfy the following condition:

d_min ≥ 2t + 1

Any code which is capable of detecting the ‘t’ errors should satisfy the following condition:

d_min ≥ t + 1

statement E is true.

Answer.1. (38/4) bits/sec

Explanation

Entropy: The average amount of information is called the “Entropy”.

$H = \;\mathop \sum \limits_i {P_i}{\log _2}\left( {\frac{1}{{{P_i}}}} \right)\;bits/symbol$

Rate of information = r.H

Calculation:

Given: r = 16 outcomes/sec

$P_1=\frac{1}{2}$, and $P_2=\frac{1}{64}$

$ H = \frac{1}{2}{\log _22} + \frac{1}{64}{\log_264};$

$H=\frac{19}{32} \ bits/outcomes$

∴ Rate of information = r.H

R_s = 16 x 19/32

R_s = 19/2 or 38/4 bits/sec

Answer.3. 2 bit

Explanation

Information associated with the event is “inversely” proportional to the probability of occurrence.

Mathematically, this is defined as:

$I = {\log _2}\left( {\frac{1}{P}} \right)\;bits$

P and I represent the probability and information associated with the event

Calculation:

With P = 1/4, the information associated with it will be:

$I = {\log _2}\left( {\frac{1}{1/4}} \right)\;bits$

$I = {\log _2}\left( {4} \right)\;bits$

I = log₂ (2²) bits

Since log_x yⁿ = n log_x y, the above can be written as:

I = 2 log₂(2)

I = 2 bits

Answer.3. 66.47 kbps

Explanation

The capacity of a band-limited AWGN channel is given by the formula:

$C = B~{\log _2}\left( {1 + \frac{S}{N}} \right)$

C = Maximum achievable data rate (in bits/sec)

B = channel bandwidth

Signal Noise power (in W) = S/N

Given

B.W. = 8 kHz, S/N = 25 dB

Since the S/N ratio is in dB,

$10~log_{10}(\frac{S}{N})=25$

$log_{10}(\frac{S}{N})=2.5$

$\frac{S}{N}=10^{2.5}~W=316.22~W$

The channel capacity will be:

$C = (8×10^3)~{\log _2}( {1 + 316.22})$

C = 66.47 kbps

Answer.3. Parity checking

Explanation

Vertical redundancy check (VRC) is an error-checking method used on an eight-bit ASCII character.

In VRC, a parity bit is attached to each byte of data, which is then tested to determine whether the transmission is correct.

Vertical redundancy check (VRC) is an error-checking method used on an eight-bit ASCII character.

Answer.1. Bits per channel

Explanation

Bits per channel states the channel capacity C, meaning the theoretical highest upper bound on the information rate of data that can be communicated at an arbitrarily low error rate using an average received signal power S through an analog communication channel that is subject to additive white Gaussian noise (AWGN) of power N.

The capacity of a band-limited AWGN channel is given by the formula:

$C = B{\log _2}\left( {1 + \frac{S}{N}} \right)$

C = Maximum achievable data rate with units of bits/sec for continuous nature of input and output. For discrete nature of input/output, bits/channel is used.

B = channel bandwidth

Signal Noise power (in W) = S/N